Dogway's Filters Packs

[Dogway]

Dogway's Filters Packs


GitHub (https://github.com/Dogway/Avisynth-Scripts) repo.

If you are new to AviSynth, to get things up and running refer to this post (https://forum.doom9.org/showthread.php?p=1986223).

TIP: For AviSynth+ front end GUIs such as MeGui/RipBot, follow the next suggestions to update to latest AVS+ version; for MeGui (https://forum.doom9.org/showthread.php?p=1964215) and RipBot (https://forum.doom9.org/showthread.php?p=1964695).

----

As others before I thought on creating one single thread to list and explain my updated filter packs and avs+ modernization efforts.

The main goal of the updates is to reduce redundant, outdated or slow functions to modern alternatives often with more features, like HBD support (32-bit float inc.), frame properties and improved performance among others.
Basic building block functions like those in masktools2, RgTools and smoothadjust have been replaced with internal Expr() wrappers, this allows liquid and easily editable code for others to inspect, debug or branch.
Additionally higher level filters have also been created or ported like those in SharpenersPack or GradePack (read below).
For performance reasons many expressions have also seen major refactors so due to this and modern AVS+ syntax updates probably any version earlier than v3.7.3 won't work properly, but this is a necessary evil to move things forward and make HBD filtering of HD|UHD sources something that is counted in hours and not days. Special thanks to pinterf for his continuous work on AVS+.


ORIGINAL

ExTools: Wrapper library for Expr() expressions that covers (and expands) most masktools2 and removegrain functions, including lutspa and convolutions. Also adds Array helper functions that expands those included internally and function approximations like 'atan', 'expr'... faster than internal. Syntax and arguments are kept so it's easy to update old scripts to the new counterparts. This pack will be required for all the following scripts.
Post about STTWM() (https://forum.doom9.org/showthread.php?p=1958299) (and initial release (https://forum.doom9.org/showthread.php?p=1950095)), Adaptive Threshold (https://forum.doom9.org/showthread.php?p=1951979), ex_bilateral() (https://forum.doom9.org/showthread.php?p=1951322) (vs Dither_bilateral16() ), ex_shape() (https://forum.doom9.org/showthread.php?p=1951609) GIF. Morphological mask filtering (https://forum.doom9.org/showthread.php?p=1950472).

Transforms Pack (https://forum.doom9.org/showthread.php?p=1979592): Divided in 3: Main, Models, Transfers. Modern color and tone response technical transforms functions for color managing AviSynth+. Goal is usability, functionality and accuracy, works over any bitdepth, supports any luma range, extra color spaces and color models among them real RGB based HSV, reversible YUV and YCoCg, IPT, OkLab (https://forum.doom9.org/showthread.php?p=1957754), ICtCp, IPTPQc2 and more. Includes also a SoftLimiter() (https://forum.doom9.org/showthread.php?p=1957754) and building block matrix functions.
Example (https://forum.doom9.org/showthread.php?p=1987349) converting an ACEScg exr to Rec709-1886 with gamut compression, tonemapping and filmic contrast.
Example (https://forum.doom9.org/showthread.php?p=1987993) converting a Dolby Vision IPTPQc2 (DVp5 or DVp8) clip to Rec709-1886 with gamut compression and tonemapping.

Grade Pack: Look transforms. Includes ex_levels() with native HBD support (same usage than native Levels() ), ex_autolevels() (https://forum.doom9.org/showthread.php?p=1982586), ex_contrast() (https://forum.doom9.org/showthread.php?p=1960722), ex_blend(), ex_glow() (https://forum.doom9.org/showthread.php?p=1948447), ex_posterize(), greyscale_rgb(), FindTemp(), WhitePoint(), Vignette(), Skin_Qualifier() (https://forum.doom9.org/showthread.php?p=1960064), GamutWarning() (https://forum.doom9.org/showthread.php?p=1960722), PseudoColor() (https://forum.doom9.org/showthread.php?p=1961307), GreyWorld() (https://forum.doom9.org/showthread.php?p=1962257), HSVxHSV() (https://forum.doom9.org/showthread.php?p=1960722) and ex_vibrance(), a saturation and vibrance function.

SMDegrain: Simple MDegrain Mod (https://raw.githack.com/Dogway/Avisynth-Scripts/master/SMDegrain/SMDegrain.html). Easy to use, fool proof degraining wrapper of MDegrain and company. Initially a small few lines wrapper by Caroliano that I took over and implemented YUY2 support, interlaced support, 16-bit dither support, contrasharpening, prefiltering, debug view, documentation, globals, and good practice code. Later on real.finder took over and adapted it to modern code, added bugfixes, HBD support and so on, so forth. Now I ported it to ExTools, sanitized the code, removed old avisynth support, Dither support, and YUY2 support and included some new features like alternative degrainers (https://forum.doom9.org/showthread.php?p=1953018), recursion (https://forum.doom9.org/showthread.php?p=1958471), low frequency restoration (https://forum.doom9.org/showthread.php?p=1956647) (here (https://forum.doom9.org/showthread.php?p=1954413) too), DCT flicker (https://forum.doom9.org/showthread.php?p=1956647), optimized UHD performance, ex_DGDenoise() and ex_BM3D() prefilters, and multi-scale retinex (https://forum.doom9.org/showthread.php?p=1953018).
Here (https://forum.doom9.org/showthread.php?p=1958459) some explanations on iterative temporal filtering. In this post (https://forum.doom9.org/showthread.php?p=1959080) a draft for a SAD sampler. Recommendations (https://forum.doom9.org/showthread.php?p=1957879) for heavy grain prefiltering. 16mm film restoration (https://forum.doom9.org/showthread.php?p=1982586) (+here (https://forum.doom9.org/showthread.php?p=1984055)). Original 2011 SMDegrain thread (https://forum.doom9.org/showthread.php?p=1547013).

Resizers Pack: Pack of functions involving resizing operations, like deep_resize() (https://forum.doom9.org/showthread.php?p=1962855) (and here (https://forum.doom9.org/showthread.php?p=1966226)) a refactored port of nnedi3_resize16() six times the speed and at higher quality, nnedi3resize(), a nnedi3 based arbitrary size scaler, RatioResize() which can resize by a single factor; percentage, adjust to width/height, to PAR, DAR and so on. PadResize() to crop or pad given input dimensions, PadBorders() (https://forum.doom9.org/showthread.php?p=1955036) like an advanced AddBorders()+Crop() with option to mirror, dilate or fill borders, MatteCrop() (https://forum.doom9.org/showthread.php?t=182678) to automatically fix (crop+resize) movies with random bordered shots, and some utilities like mmod() to crop/pad/auto to mod, and nmod() (https://forum.doom9.org/showthread.php?p=1955036) to mod values with extra features like min value or bankers' rounding.

Masks Pack: Mask and limiter filters. BoxMask(), FlatMask() (https://forum.doom9.org/showthread.php?p=1959356), LumaMask(), CornerMask() (https://forum.doom9.org/showthread.php?p=1964580) (cheap alternative here (https://forum.doom9.org/showthread.php?p=1969058)) and MotionMask() for masks and ex_limitchange(), ex_limitdif() and Soothe() for limiters.

Scenes Pack: SceneStats() (https://forum.doom9.org/showthread.php?p=1967432) opens the doors to scene based workflows. It writes current (frame) scene range bounds into '_SceneRange' frame properties and current scene change into '_SceneChangePrev', also scene motion into '_SceneMotion', scene details to '_SceneDetails' (a complexity index for average of edges), scene exposure index to '_SceneExposure' and pixel stats into '_SceneStats', on the fly or by offloading it to a file. ReadStats() can load an optionally exported SceneStats() stats file for faster processing at encoding stages. ClipStats() will otherwise load them and convert them to clip global stats, to help you decide better clip-wide constant settings in your filters.
Example for SMDegrain (https://forum.doom9.org/showthread.php?p=1969107).
Example for FilmGrainPlus (https://forum.doom9.org/showthread.php?p=1985953).

FilmGrain+: Made from the ground up, an accurate and performant synthetic film grain filter with presets for the most common negative films.

Logo: Easily add static logos or watermarks, with blur, fade in/out, opacity, and blending controls. Eventually also for video based logos.

Stabilization Tools Pack (legacy): Initially a simple mod of Stab() which grew bigger and currently includes various strategies for edge filling. Also includes FilmGateFix() mainly aimed at anime sources.


EX/MIX MODS

Normally there are 2 flavors of each mod:
EX mods are future proof with ExTools wrappers and minimal dependencies, this can also come handy when running on Linux/macOS which very few plugins support.
MIX mods use carefully chosen masktools2 and removegrain functions to maximize speed but come with these and probably other dependencies as well.

QTGMC+ (https://forum.doom9.org/showthread.php?t=174544): Reference deinterlacer. Ported (https://forum.doom9.org/showthread.php?p=1948549) to ExTools from v3.382 (~40% faster in HBD). Includes ex_vinverse() (now legacy for int bitdepths), ex_bob() and ex_reduceflicker() functions.

LSFplus (https://forum.doom9.org/showthread.php?t=142706): Based on LSFmod, also one of the best sharpeners out there. Optimized (+74% with no SS), ported to ExTools and added more features.

GradFun3plus: Port of cretindesalpes' excellent GradFun3 debanding filter to internal AVS+ calls (+66% gain in smode=0)

Sharpeners (https://forum.doom9.org/showthread.php?p=1948775) Pack (https://forum.doom9.org/showthread.php?p=1951536): Collection of high quality sharpeners optimized and ported to ExTools for HBD support and performance. In total 29 sharpeners, among them; Adaptive Sharpen, ex_unsharp, CASP, NVSharpen (https://forum.doom9.org/showthread.php?p=1958035), ex_ContraSharpening, SeeSaw, FineSharpPlus, NonlinUSM, ReCon, blah and Plum.

Deblock Pack (https://forum.doom9.org/showthread.php?p=1952485): Pack containing different deblocking functions from famous Deblock_QED() (29% speed gain), to CCD(), SmoothD2c(), SmoothDeblock() (WIP) or feisty2's Oyster (Oyster includes also deringing and else)

Similarity Metrics: Pack containing all the similarity/distance metrics ported by Asd-g to AVS+ from WolframRhodium VapourSynth repo. I collected, sanitized and updated the code for x4 speed gain on GMSD(), x2 on MDSI(), x3 on vsSSIM() and added+refactored BSSIM() from zorr. Also created SVM(), a metric for image sharpness. For more metrics check the cost functions in ex_makediff() (https://forum.doom9.org/showthread.php?p=1955541) in ExTools.

yugefunc (https://forum.doom9.org/showthread.php?p=1951536): Collection of VapourSynth filters ported to AVS+ and optimized on the way with ExTools and other expression tricks: ex_guidedblur(), ex_ANguidedblur(), XDoG() (WIP), etc

Other: Some other scripts have received the ExTools treatment; FillMissing(), FastLineDarkenPlus(), SPresso(), DeStripe() (https://forum.doom9.org/showthread.php?p=1950857), etc


ExTools main functions:
# EXPRESSIONS
ex_lut() - Single variable (1 clip) expressions
ex_lutxy() - Double variable (2 clips) expressions
ex_lutxyz() - Triple variable (3 clips) expressions
ex_lutxyza() - Quadruple variable (4 clips) expressions
ex_makediff() - Clip based differentiation. Also calculates similarity/residual metrics via cost functions
ex_adddiff() - Sum clips, specially useful to add back the result of differentiation
ex_makeadddiff() - ex_makediff() and ex_adddiff() in one step
ex_logic() - Logical operations between 2 clips with logic ops (MIN, MAX, OR, AND, etc)
ex_merge() - Merging. Performs a linear interpolation between 2 clips based on mask (3rd clip)
ex_clamp() - Clamps first clip between the maximum of the second clip and the minimum of the third
ex_binarize() - Performs binary type segmentation or thresholding
ex_athres() - Adaptive Threshold. Special binary thresholding for uneven brightness images (ie. extracting letters from a shaded area)
ex_invert() - Invert the clip pixel values
ex_lutspa() - Relative or absolute pixel-location based expressions
ex_motion() - Computes a very primitive motion mask akin to MaskTools2's mt_motion()
ex_hysteresis() - Proof of concept Expr() port of mt_hysteresis(). Uses 'for' loops so very slow

# MORPHOLOGICAL
ex_expand() - Morphological dilation/expansion of pixel-value based on structuring element given by the kernel window
ex_inpand() - Morphological erosion/contraction of pixel-value based on structuring element given by the kernel window
ex_inflate() - Expansion via outward blurring given structuring element of pixel values of the kernel window
ex_deflate() - Contraction via inward blurring given structuring element of pixel values of the kernel window
ex_hitormiss() - Structuring elements based morphological transforms for binary images
ex_edge() - Gradient magnitude. Edge detection via (partial) local derivatives
ex_luts() - Moving window relative pixel-location based expressions. A convolution do-it-all filter
ex_shape() - Helper filter for ex_luts() (and other expression based filters) to fetch kernel-window pixels into a string

# BLURS
ex_boxblur() - Discreet local neighborhood blur convolutions
ex_blur() - Gaussian (or Butterworth) weighted blur convolutions
ex_gaussianblur()- Optimized Gaussian filter for large sigma
ex_kawase() - Kawase optimized blur filter (still slower than ex_gaussianblur() ). Accepts different strides so good for exponential blur
ex_blur3D() - Spatio-temporal blur filter
ex_bilateral() - Bilateral blur filter (respects edges)
ex_smartblur() - Like Bilateral filter but more performant (mimics Photoshop's Surface Blur)
ex_smooth() - Savitzky-Golay smoothing filter. Halfway between blur and antialiasing
ex_FluxSmoothT() - Minimum change between a temporal weighted blur and temporal median. Informal port of FluxSmoothT filter via Didée's description
ex_FluxSmoothST()- Spatio-Temporal minimum change between weighted blur and median. Uses ex_FluxSmoothT() and its spatial equivalent ex_MinBlur()
ex_median() - Median (rank order) based blur filtering. Also includes some alternative mean average algorithms
ex_repair() - Median (rank order) based repair filter
STTWM() - Spatio-Temporal Thresholded Weighted Median (STPresso() inspired / not a port)

[Dogway]

Benchmarks for 1080p 16-bit

ExTools

DGSource("1080psrc.dgi")
ConvertBits(16)

# mt_makediff() is 5% slower than ex_makediff()
# mt_adddiff() is 5% slower than ex_adddiff()
# mt_logic(mode="and") is 6% slower than ex_logic(mode="and")
# mt_merge(luma=true,U/V=3) is 8% faster than ex_merge(luma=true, UV=3) (5% slower when Y clips/masks)
# mt_clamp() is 6% slower than ex_clamp()
# mt_binarize() is 12% slower than ex_binarize()
# mt_invert() is 10% slower than ex_invert()
# invert(channels="Y") is 17% slower than ex_invert()
# mt_lutspa() is 0% slower than ex_lutspa()
# mt_luts()* is 96% slower than ex_luts() *tested with mt_luts( c, mode="max", pixels=mt_square( 1 ), expr="x y - abs")
# Overlay(mode="multiply") is 44% slower than ex_blend(mode="multiply")
# Overlay_MTools(mode="multiply") is 8% slower than ex_blend(mode="multiply")
# OverlayPlus(a, mode="multiply") is 1% slower than ex_blend(mode="multiply")

# mt_expand() is 6% faster than ex_expand()
# mt_inpand() is 5% faster than ex_inpand()
# mt_deflate() is 14% faster than ex_deflate()
# mt_inflate() is 14% faster than ex_inflate()
# mt_edge()* is 14% faster than ex_edge() *but much slower in "free" kernel mode

Prefetch(4)

In any case there are a few exceptions, for example mt_expand(mode= mt_circle(zero=true, radius=2)) is much much slower than ex_expand(2,mode="circle"), and in the same vein ex_expand(2) could be same speed or faster than mt_expand().mt_expand().

# Bilateral Blur
#
# 100% Dither_bilateral16(radius=2, thr=10, flat=1.0, u=1, v=1) (216fps) # Output is dirtier though
# 77% ex_bilateral(1,dejaggie=false)
# 59% vsTBilateral(diameterY=3, sdevY=4, idevY=4.0, u=1, v=1)
# 43% TBilateral(3,3,chroma=false) # only supports 8-bits
# 23% bilateral(sigmaSY=1, sigmaRY=0.02, algorithmY=2, u=1, v=1)

Prefetch(6)

# Variable Box Blur
#
# 100% removegrain(20,-1) (485fps)
# 91% ex_boxblur(1,mode="mean",UV=1)
# 90% MiniDeen(radiusY=1, thrY=255, u=1,v=1) # crumbles from rad=3 onwards
# 89% neo_MiniDeen(radiusY=1, thrY=255, u=1,v=1)
# 80% mt_inflate().mt_deflate() # mean blur approximation
# 70% ex_blur(1.5,n=300,mode="butterworth")
# 69% blur(1.58)
# 67% generalconvolution(matrix="1 1 1 1 1 1 1 1 1",chroma=false)
# 65% Dither_box_filter16(2,U=1,V=1) # with ConverttoStacked() and ConvertfromStacked()
# 44% mt_convolution("1 1 1","1 1 1",U=1,V=1)
# 13% SpatialSoften(1,30,0) # 8-bit YUY2 only, thresholded. Prefetch(8)
# 5% mt_luts(last, "avg", mt_square(1), "y",chroma="-1")

# Variable Gaussian Blur (binomial fitted)
#
# 100% removegrain(12,-1) (486fps) # technically a binomial weighted mean of [1 2 1]
# 97% GBlur2(sqrt(1)/2. * sqrt(2),chroma=2) # only in 8-bit. weighted mean of [1 2 1]
# 90% ex_boxblur(1,mode="weighted",UV=1) # binomial weighted mean
# 88% ablur(1, 1, chroma=1) # against ex_boxblur(2,mode="weighted",UV=1)
# 85% BinomialBlur(sqrt(1)*0.707,U=1,V=1) # only in 8-bit
# 81% vsTCanny(sqrt(1)*0.707,mode=-1,u=1,v=1) # true gaussian blur (fastest for mid size sigma)
# 70% ex_blur(1,mode="binomial,UV=1) # true gaussian blur
# 68% blur(1.00) # weighted mean of [1 2 1]
# 65% generalconvolution(matrix="1 2 1 2 4 2 1 2 1",chroma=false)
# 44% mt_convolution("1 2 1","1 2 1",U=1,V=1)
# 23% GBlur(rad=1,sd=0.9,u=false,v=false)
# 11% FastBlur(sqrt(1)*0.707,gamma=false)
# 11% GaussianBlur(0.53,U=1,V=1) # only in 8-bit

Prefetch(4)



Script mods

CPU: i7-4790K (Stock Clock)
GPU: GTX 1070

Prefetch(6)
LSFmod.v2.193: 45.4fps
LSFmod.v6.0ex: 60.0fps
LSFplus.v6.0mix: 61.8fps
LSFplus(preset="slow",strength=200,edgemode=0,soothe=true,ss_x=1.0,ss_y=1.0)


Prefetch(4)
GrainFactoryLite: 57fps (96fps @8-bit)
GrainFactory3mod EX: 68fps (84fps @8-bit) Prefetch(8)
FilmGrain: 52fps (75fps @8-bit) Prefetch(8)
FilmGrain+: 38fps (54fps @8-bit) ('gamma' mode)
FilmGrain+: 37fps (50fps @8-bit) ('log' mode)
str=1.25
size=1.2
GrainFactory3mod(size=1,g1str=6.0*str,g2str=8.0*str,g3str=5.5*str,g1size=1.20*size,g2size=1.50*size,g3size=1.40*size,g1cstr=0.9,g2cstr=0.9,g3cstr=0.9,temp_avg=1)
or
FilmGrain(size=1.1,str=9,cstr=0.5,coarse=4.0,conv=false)
or
FilmGrainPlus(size=1.5,str=0.8,lo=1,mid=1,hi=1,sharpness=0.8,mode="gamma")

Prefetch(6)
SMDegrain v3.1.2.111s: 3.300fps (slowdown due to Contrasharpening() )
Prefetch(8)
SMDegrain v4.3.0d: 15.1fps
SMDegrain(tr=2,thSAD=400,contrasharp=true,refinemotion=true)


Note: QTGMC+ v4.0 is the last version comparable to older ones. Later versions use different default core deinterlacers for higher detail preservation.

Prefetch(8) (720x576 clip)
QTGMC 3.382s: 23 fps (8-bit) 10.0 fps (16-bit)
QTGMC+ 4.00p: 22 fps (8-bit) 18.0 fps (16-bit)
QTGMCp(tr2=3,preset="very slow",Lossless=2,sourcematch=3,sharpness=0.2,MatchEnhance=0.0,MatchPreset="Slow", MatchPreset2="Slow",border=true,threads=4)
QTGMC 3.382s: 54.7 fps (8-bit) 31.5 fps (16-bit)
QTGMC+ 4.00p: 55.4 fps (8-bit) 40.0 fps (16-bit)
QTGMCp(thsad1=300,blocksize=8,TR0=1,TR1=1,TR2=0,EZKeepGrain=1.0,NoiseDeint="Generate",StabilizeNoise=true,border=true,chromamotion=false,threads=4)
QTGMC 3.382s: 89 fps (8-bit) 42.1 fps (16-bit)
QTGMC+ 4.00: 87 fps (8-bit) 60.0 fps (16-bit)
QTGMCp(tr2=2,preset="slow",border=false,threads=4)
QTGMC 3.382s: 15.4 fps (8-bit) 12.3 fps (16-bit)
QTGMC+ 4.00p: 13.2 fps (8-bit) 12.3 fps (16-bit)
QTGMCp(tr2=2,preset="very slow",SVThin=0.5,EZKeepGrain=2.0,NoisePreset="slower",Sharpness=0.7,tuning="DV-SD",border=true,threads=4)
QTGMC 3.382s: 72.6 fps (8-bit) 38 fps (16-bit)
QTGMC+ 4.00p: 73.3 fps (8-bit) 53.5 fps (16-bit)
QTGMCp(TR2=3,TR0=1,TR1=1, Preset="Slower", InputType=1, sharpness=0)



ex_median(), ex_bilateral() with Prefetch(6)
100.0% vertical Prefetch(4) (440fps)
97.5% undot Prefetch(4)
97.0% undot6 Prefetch(4)
96.8% cartoon Prefetch(4)
94.3% edgeS Prefetch(4)
91.1% verticalS Prefetch(4)
90.5% medianT Prefetch(4)
84.5% median
84.3% SixNN Prefetch(4)
82.2% PML
81.8% edgeC
80.9% undot3 Prefetch(4)
80.7% undot2 Prefetch(4)
80.7% midsum
79.8% EMF
77.0% medianT5 Prefetch(4)
76.7% GaussT5
76.6% IQM
75.2% ML3D
75.0% edgeW
75.0% winsor
73.4% trimean
72.0% edgeCL
71.4% smart
70.5% SNN
70.2% CAM
69.1% CWM
67.6% CWM2
66.5% AWM
52.6% MMF
52.2% PWM
49.3% WMF
48.2% IQMST
44.1% ML3Dex
38.0% bilateral
36.1% Hybrid
33.2% STWM
33.2% kuwahara
31.1% BDM
29.0% unblob3D
28.9% DGM5
28.1% TL3D
26.4% DGM3
26.0% DGM2
25.9% unblob3
25.9% DGM1
25.7% DGM4
25.5% median5
24.8% DGM0
21.5% medianST
20.2% trimean5
19.9% median7o
19.1% smart2
18.5% AMF
18.2% IQM5
17.3% winsor5
16.8% medianSTS
16.1% IQMV
16.1% GaussST5
8.1% median7 Prefetch(8)
6.3% smart3 Prefetch(8)
ex_median() comparison chart.................................source (for toggle comparison)
http://i.imgur.com/CU8LIWdm.jpg (https://i.imgur.com/CU8LIWd.jpg)..........http://i.imgur.com/SUokG79m.jpg (https://i.imgur.com/SUokG79.jpg)


ex_blur(), ex_blur3D(), ex_boxblur(), ex_smooth(), ex_kawase()
100.0% rg19 (448fps)
99.1% bokeh2
98.7% kawase lin
97.8% weighted
97.8% mean
86.4% kawase2 lin
86.2% bokeh
78.3% SNN
77.0% rg192
75.9% mean2
75.7% smooth
75.4% weighted2
75.4% blur
72.8% smartblur
71.9% smooth2
71.4% smooth sharp
71.4% blur2
67.0% smooth2 sharp
64.3% smartblur2
60.7% trimmed
60.7% weighted3D
52.5% mean3D
37.1% ex_fluxsmoothST
blurs comparison chart............................................source (for toggle comparison)
http://i.imgur.com/1La22M9m.jpg (https://i.imgur.com/1La22M9.jpg)..........http://i.imgur.com/61WVEcVm.jpg (https://i.imgur.com/61WVEcV.jpg)


ex_edge() with default thresholds and Prefetch(4)
100.0% mt_sobel (460fps)
99.3% tritical
97.0% cartoon
96.3% hotdog
95.0% kayyali
94.6% laplace
91.1% hprewitt
89.8% SGDD
89.1% min/max
89.1% sobel5
88.3% roberts
88.0% max
87.0% qprewitt
87.0% LoG
86.5% TEdge
85.2% frei-chen
84.9% kroon
84.6% prewitt
84.1% sobel
84.1% farid
84.0% pscharr
83.3% scharr
81.7% robinson
79.8% SGDD7
78.9% DoG Prefetch(6)
71.3% Std Prefetch(6)
62.2% kirsch Prefetch(6)
56.0% DoB Prefetch(6)
50.4% farid5 Prefetch(6)
49.8% SG Prefetch(6)
47.4% FDoG Prefetch(6)
ex_edge() comparison chart
http://i.imgur.com/NgoDbttm.png (https://i.imgur.com/NgoDbtt.png)


Sharpeners Pack
360 100 % XSharpenPlus()
355 98.6% CASP(1)
347 96.4% UnsharpMask_HBD(128*n,1,0) Prefetch(4)
339 94.2% DGSharpen2()
317 88.1% ex_unsharp() Prefetch(4)
310 86.1% DetailSharpen()
272 75.6% NonlinUSM()
250 69.5% FineSharpPlus()
178 49.4% pSharpen()
160 44.4% RSharpen()
150 41.7% LSFplus(preset="LSF")
148 41.1% CASm()
145 40.3% SharpenComplex2()
106 29.4% NVSharpen() Prefetch(8)
97 26.9% ex_ContraSharpening(a)
79 21.9% SlopeBend()
60 16.7% LSFplus(preset="fast")
50.6 13.1% DelicateSharp()
43.2 12.0% LSFplus(preset="medium")
33 9.2% SSSharpFaster()
27 7.5% LSFplus(preset="slow")
26.5 7.4% SeeSaw(a)
24.9 6.9% ReCon()
21 5.8% MedianSharp()
18.5 5.1% Adaptive_sharpen(1.0) Prefetch(8) (32-bit)
14.5 4.0% MedSharp()
11.7 3.3% blah() Prefetch(4)
1.9 0.5% SSSharpEX() Prefetch(4)
0.22 0.06% RegularSharp()

[pinterf]

Dogway's Filters Packs
* Expr() is known to perform worse in 8-bits than masktools2 and sometimes even in 16-bit. This is due to lack of lut based calculations in 8-bit, and lack of AVX2 acceleration for convolutions ("pixel addressing"), so work here is anticipating future performance improvements on avs+.

I'm so glad that finally someone is using pixel-addressing. I've done it only for my pure curiousity. The acceleration part took me many weeks to implement; it became so complex that I wanted to remove it. Like in a game theory: when will I lose more if I'm giving it up and drop two week's work or if I'm working on it further :)

I have to mention that no AVX2 code is used when pixel addressing is used in the Expr expression due to complexity of the implementation code for AVX 32 byte registers.

Expr based basic luts in Avisynth: they are on my roadmap, there are hints in the my source already I'm planning to continue the work on that topic later.

Masktools2 is using internally 64 bit doubles while Expr is using only 32 bit floats.

[kedautinh12]

Thank pinterf

[real.finder]

I'm so glad that finally someone is using pixel-addressing. I've done it only for my pure curiousity. The acceleration part took me many weeks to implement; it became so complex that I wanted to remove it. Like in a game theory: when will I lose more if I'm giving it up and drop two week's work or if I'm working on it further :)

I have to mention that no AVX2 code is used when pixel addressing is used in the Expr expression due to complexity of the implementation code for AVX 32 byte registers.

Expr based basic luts in Avisynth: they are on my roadmap, there are hints in the my source already I'm planning to continue the work on that topic later.

Masktools2 is using internally 64 bit doubles while Expr is using only 32 bit floats.

I did use it before if it this https://github.com/realfinder/AVS-Stuff/blob/7965aa0d1bd7aeb1dd3639ff48e09bb982acf92d/avs%202.5%20and%20up/AdvancedDenoising.avsi#L447 also to imitation https://web.archive.org/web/20150602063631/http://home.earthlink.net/~tacosalad/video/dotcrawl.html in here (https://forum.doom9.org/showpost.php?p=1942945&postcount=16)

[Dogway]

pinterf: Thanks to you. Very useful tools, also the array implementation. I do think these new features are under-utilized as I could see so I put them in good use in some of the packs, sanitizing every convoluted script I can find.

I know that for some doing this in avs scripting might look foolish, but IMO it democratizes the code, makes it more liquid and promotes avs+ development. That's why while it might currently underperform in certain situations (lutspa, pixel addressing, etc) it levels out with the improvement in the Expr expressions, as can be seen in the benchmarks. It can only get better, I hope.

Yes, I saw mt_xxpand got AVX2 a year ago, while Expr is on SSSE3 for pixel addressing. I have no programming skills but I can imagine, mainly as Expr is much more powerful hence harder to optimize.

[Dogway]

So I've been working like mad on ExTools and it's almost reaching v1.0 final. Only ex_edge() is left from my planned ports (and ex_clamp which isn't still a 1:1 replica)

Today I added kernel iterated gaussian and box blur functions as separated 2x1D kernels. They run pretty fast, I'm just wondering if I should add a multiplier/divisor to make sigma stepness more granular.

Later I also want to optimize the other kernels in case they are separable and make iterators for them to make radius work properly. Since I already made them for blurs it should be easier to port. Once done I switch back to Transforms Pack for v1.0 final.

On the long run I want to fiddle with DotCrawl convolutions, add more edge detection kernels, and make some Unsharps with them.

Expr based basic luts in Avisynth: they are on my roadmap...

Thanks for the work, looking forward it, there are many situations where HBD isn't needed like frame interpolation or deinterlacing.

Masktools2 is using internally 64 bit doubles while Expr is using only 32 bit floats.

Does that make a difference in performance? Haven't run tests without use_expr>0

[Dogway]

ExTools (https://github.com/Dogway/Avisynth-Scripts/blob/master/ExTools.avsi) v1.0 final is released. Now it also supports 32-bits float bitdepth.
It's generally faster than masktools2 except when Expr() "pixel addressing" feature is used like in convolutions.
For 8-bit it's still slower than masktool2, but pinterf is currently working on it.

I don't know if "pixel addressing" is ever going to have AVX2 acceleration, but if it does it might possible for my ports to exceed masktools2 speed, or at least reach it, leaving masktools2 dependency behind and just work with internal code, as ideally it should be.

Aside from masktools counterparts I also created a few functions like ex_blend() which replaces Overlay(), ex_undot() for removegrain(1), ex_boxblur() for removegrain(19) and ex_blur() for removegrain(12) and blur().

Therefore for my script mods there are two versions divided in folders; EX mods, and MIX mods. EX mods are future proof with ExTools wrappers, and MIX mods use masktools2 convolutions and removegrain to maximize speed.

See updated benchmarks in second post (https://forum.doom9.org/showthread.php?p=1943156#post1943156).

[kedautinh12]

Thanks for hardwork Dogway, but I think you just focus EX mods and MIX mods we can use real.finder's scripts with same your

[Dogway]

As you can see on the benchmarks many functions are faster than masktools2 calls in real.finder mods. Specially comes to mind Overlay() which is uber slow, but I also do some optimizations aside from 1:1 ports. I don't plan to port everything, just my most used scripts so I take special care. I'm also cleaning the code, removing old compatibility support, formatting, and so on.

I might replace ex_merge() back to mt_merge() for MIX mods, something is going on in there, but didn't have much time to debug.

From now I will resume TransformsPack to release a v1.0 final soon, focused on SDR color spaces.

[pinterf]

Note: mt_merge has a cplace parameter default "mpeg2" which - with luma = true - is slower than the dumb "mpeg1" choice. Could you try your benchmarks wih cplace="mpeg1" ? Regarding the other benchmarks, I'll do them as well, for example why mt_binarize is slower.
EDIT: Overlay multiply (largest speed difference): no wonder, there is no SIMD optimization there at all.
EDIT2: mt_invert and Avisynth Invert is SSE2 only. But there is only a single instruction or two between load and store which usually implies no or little gain.
Actually some years ago I've implemented for example 8 bit binarize functions in AVX2 but I got zero speed gain so I decided that it won't go live yet. Time to test those again on my i7-7700.

[GMJCZP]

As you can see on the benchmarks many functions are faster than masktools2 calls in real.finder mods. Specially comes to mind Overlay() which is uber slow, but I also do some optimizations aside from 1:1 ports. I don't plan to port everything, just my most used scripts so I take special care. I'm also cleaning the code, removing old compatibility support, formatting, and so on.


DogWay:

Do you have a function equivalent to Overlay, but optimized?

[kedautinh12]

Here:
https://github.com/Dogway/Avisynth-Scripts/blob/392da5af3d39529dc125de658fe841a787c62a68/ExTools.avsi#L149

[pinterf]

DogWay:

Do you have a function equivalent to Overlay, but optimized?
Overlay is basically many different filters under a common name.
Looking at the code: only 'blend', 'lighten' and 'darken' are optimized.
When there is a popular and frequently used mode _and_ affects scripts significantly with its slowness, probably I can implement a speedup.

[Reel.Deel]

VapourSynth's havfunc's Overlay script (https://github.com/HomeOfVapourSynthEvolution/havsfunc/blob/master/havsfunc.py#L5427) (which if I'm not mistaken, mimics AviSynth's Overlay) uses Expr and MaskedMarge to do the work. It can probably be translated into AviSynth easily, it even includes some additional modes not available is AviSynth's Overlay.

[pinterf]

Note: mt_merge has a cplace parameter default "mpeg2" which - with luma = true - is slower than the dumb "mpeg1" choice. Could you try your benchmarks wih cplace="mpeg1" ? Regarding the other benchmarks, I'll do them as well, for example why mt_binarize is slower.
EDIT: Overlay multiply (largest speed difference): no wonder, there is no SIMD optimization there at all.
EDIT2: mt_invert and Avisynth Invert is SSE2 only. But there is only a single instruction or two between load and store which usually implies no or little gain.
Actually some years ago I've implemented for example 8 bit binarize functions in AVX2 but I got zero speed gain so I decided that it won't go live yet. Time to test those again on my i7-7700.

I was checking the issue with mt_binarize benchmarks, because the processing itself is more processor-heavy when using Expr and I did not understand, why it is still slower.

The common in mt_binarize and Expr-based ex_binarize that they read and store pixels.

What they are doing inside:

mt_binarize (16 bit data) has 2 operations:
- integer addition
- comparison.

Expr:
- Converts 16 bit pixels to 32 bit float (size doubled, using two register instead of one)
- Compares with the limit (float comparison)
- Mask-blends either 0.0f or 65535.0f depending on the result.
- Converts back float data to 16 bits integer with rounding.

Well, this difference can be seen in the single-threaded benchmark results.

Doing almost nothing, quite interestingly mt_binarize alone is so fast that we better not do any synthetic benchmark on it - and in general with such filters (like mt_logic). I recommend to test them only embedded in a real script. (Like Dogway has did as well when provided benchmarks for whole scripts)

mt_binarize is a minimal-operation filter, having a memory load + two register operations + memory store.
Clearly it was reaching the memory bottleneck.
mt_binarize with no MT(!) is even a bit quicker than with any Prefetch values. This must be due to ruined caching and task swithing/register saving overhead.

mt_binarize combined with RemoveGrain was in the same ballpark with Prefetch(4) than without RemoveGrain!

Tested on i7-7700, avs+ 3.7.work

#SetMaxCPU("SSE4.1")
Import("ExTools.avsi")
Colorbars(pixel_type = "YUV420P16")
mt_binarize()
#ex_binarize()
#RemoveGrain(1, -1)
#Prefetch(4) # 8


Data in fps, on my system the values are ~average, actual values fluctuate, but we can see the trends.

Prefetch mt_binarize ex_binarize x64_mt_bin x64_ex_bin
- 19000 7000 19100 6700
4 16000 16500 15900 16600
8 13000 13900 12600 13900


Paired with a RemoveGrain after mt_/ex_binarize:

#SetMaxCPU("SSE4.1")
Import("ExTools.avsi")
Colorbars(pixel_type = "YUV420P16")
mt_binarize()
#ex_binarize()
RemoveGrain(1, -1)
#Prefetch(4) # 8



Prefetch mt_binarize ex_binarize x64_mt_bin x64_ex_bin + RemoveGrain(1, -1)
- 8800 5000 8500 4500
4 16114 11400 16700 11200

[Dogway]

Yes, that was a test I had planned in my head because on real scripts it seems my functions perform slower while in synthetic normally is faster. I don't really understand why. One thing I thought, and that's why I asked I thought masktools2 was converting to double float, but here is 16-bit integer like mt_binarize in your example above. I would think double float has a performance penalty as you explained.

By the way are those fps in the thousands? I get 530fps but I use a bit more real case scenario (not totally synthetic), 1080p source, load with DGSource and process in 16-bit.

I crafted a small script, more like what happens within filters:

250fps
ConvertBits(16)
a=ex_binarize(68)
b=a.ex_invert()
ex_logic(a,b,mode="andn")
# removegrain(1,-1)
Prefetch(4)

215fps
ConvertBits(16)
a=mt_binarize(68)
b=a.mt_invert()
mt_logic(a,b,mode="andn")
# removegrain(1,-1)
Prefetch(4)

EDIT: By the way I had two more algos for binarize, but they were same or slower than the ternary:
str = Format("x x {th} scaleb - x {th} scaleb + clip")
str = Format("x x {th} scaleb - * ")

Enabling removegrain (or disabling Prefetch) didn't make much of a difference. I also have to test ex_merge() but the mask handling got me a bit on my nerves and it was the last thing I did when I was already burned with ExTools.

On ex_blend, I have plans to add more blending modes, same for ex_expand shapes, ex_edge modes, unsharp and so on. But wanted to get the basics first and on a later time improve the project with fresher eyes.

[pinterf]

I have Colorbars source filter, that why it is quicker.

Double (mt_lut) vs Float (Expr): the difference affects only where expressions must be evaluated Expr, and mt_lut family.
When there is enough memory and mt_lut is really using LUT then the slow calculations affect only the creation of lut tables.
But for a 16bit lutxy there is no memory for lut (we'd end with a 8GB memory table), so masktools is using 'realtime' expression evaluation. Calculates the expression for each frame and for each pixel. In pure C code.
And that is very slow.
Expr is calculating realtime as well. But since it compiles the expression into SSE2/AVX2 machine code (acts like a small compiler) it is quicker than realtime mt_lut by magnitudes.

Usually non-lut masktools filters are optimized heavily and use integer where the source is 8-16 bits.

[Dogway]

Usually non-lut masktools filters are optimized heavily and use integer where the source is 8-16 bits.

That makes sense now. I thought it only was the convolution types. I also noticed chaining removegrain (or Dither_boxfilter) is very fast, whereas Expr() suffers a lot, probably due what you explained. So processing in 32-bit float would tell another story.

[GMJCZP]

Overlay is basically many different filters under a common name.
Looking at the code: only 'blend', 'lighten' and 'darken' are optimized.
When there is a popular and frequently used mode _and_ affects scripts significantly with its slowness, probably I can implement a speedup.

Fortunately I am only interested in the blend mode, I should check out how to equate ex_blend with Overlay.

Thanks Dogway for your contribution.

[pinterf]

That makes sense now. I thought it only was the convolution types. I also noticed chaining removegrain (or Dither_boxfilter) is very fast, whereas Expr() suffers a lot, probably due what you explained. So processing in 32-bit float would tell another story.
These masktools2 filters has no simd versions, probably because they do generic stuff.
mt_convolution, mt_gradient, mt_mappedblur, mt_hysteresis

[GMJCZP]

I am trying to substitute Overlay for ex_blend in my RescueFrame (https://forum.doom9.org/showthread.php?p=1699095#post1699095) script, and regardless of whether I use 8 bits or not, I am getting the following error message:

"Expr: All inputs must have the same number of planes and the same dimensions, subsampling included."

If I use the only ConverttoYv12 that works I only get a black image.

[Dogway]

@pinterf: mt_hysteresis was one I doubted to include or not. I will have a look.

@GMJCZP: I checked it but what is it blending? it looks like blending with opacity of 1.0, basically frame replacing. There are tools already for that like ReplaceFramesSimple or Prune by StainlessS. ex_blend needs clips a and b to be the same format or it will error out.

[GMJCZP]

Don't Overlay and ex_blend have identical behavior?

[Dogway]

Updated ExTools to improve performance for ex_xxpand() by decoupling the thresholding (now just 6% shy from mt_expand() and with only SSSE3!!!) .
Also improved ex_blur() performance when rad>1 by precomputing final pixel weights.
Finally added the LUT expr family for easier porting the code and UV plane handling.

I also had time to sanitize SMDegrain, still not complete sanitation but near (v3.2.0d RC1). Removed old avs, Dither and YUY2 support. Ported masktools2 to ExTools in a "MIX" fashion and fixed a few small issues. Finally updated code to new avs+ grammar and properly formatted it.

By the way, Transforms Pack is also near to v1.0 final, it's advancing nice, 32-bit support, 2020CL and NCL conversion, and bugfixes.

[kedautinh12]

Thanks

[tormento]

I also had time to sanitize SMDegrain, still not complete sanitation but near (v3.2.0d RC1). Removed old avs, Dither and YUY2 support. Ported masktools2 to ExTools in a "MIX" fashion and fixed a few small issues. Finally updated code to new avs+ grammar and properly formatted it.
Wow!

Considering it is just the script I use plain always, big thanks!

[Dogway]

Yes, it should be a bit faster. In my tests (check post#2) it performs 0.150fps better which translates into shaving 30 mins in encoding speed for a full feature. In any case the major bottleneck in SMDegrain is mvtools itself. I also plan to add an UHD mode, to try to make it faster in such cases.

[GMJCZP]

Don't Overlay and ex_blend have identical behavior?

Your function does not accept ConvertToYuv24(), which does accept Overlay, and could help mitigate at least the effects of conversions.

[Dogway]

Sorry I don't understand well. It works with YV24 on my side, how is your script?
It's not a 1:1 port of Overlay(), my blending modes matches Photoshop's.

[GMJCZP]

Sorry I don't understand well. It works with YV24 on my side, how is your script?

In my signature.

It's not a 1:1 port of Overlay(), my blending modes matches Photoshop's.

Therein lies the problem, I cannot achieve 100% emulation. RescueFrame calls Overlay to replace one or a group of frames with an image and so far I have not succeeded with Ex_blend.

[tormento]

In any case the major bottleneck in SMDegrain is mvtools itself. I also plan to add an UHD mode, to try to make it faster in such cases.
As I wrote in other thread, I dream of MVTools2 on CUDA.

[tormento]

Yes, it should be a bit faster.
My current syntax with realfinder version is

SMDegrain (tr=4, thSAD=400, refinemotion=false, n16_out=true, mode=0, contrasharp=false, PreFilter=4, truemotion=true, plane=4, chroma=true)

to have a 16 bit output from 8 bit input.

Do you plan to introduce a similar flag or is it sufficient to delete it and add ConvertBits(16) before SMDegrain?

There is mode=0 too, about dithering, that is useless if working with a 16 bit chain. Have I to ignore it or simply use -1 as argument?

[Dogway]

Yes, ConvertBits(16) beforehand for 16-bits.

I had a look at SVPFlow but it only supports 1 frame delta, also it was slower in my tests... Maybe when mvtools2 from Neo gets ported back, but pinterf said he isn't doing it so someone else needs to be up for the task.

I'm currently working on further SMDegrain cleanup, in my internal version mode is removed also, might upload today or tomorrow since I added quite a few changes.


On another note, I have been advancing with Transforms Pack, now in RC8. Internally I also added linear_in and linear_out arguments (when dealing with linear rgb image sequences for example) and a convert format only option, so you can convert from YUV420 to YV444 without any gamma nor gamut processing, it also uses a lazy mode input, so easy to use.

[tormento]

Yes, ConvertBits(16) beforehand for 16-bits.
Would it be possible to introduce some depan/stabilize while denoising? We already have some vectors ready, perhaps it could be almost computation free.

[Dogway]

Normally you want to denoise after stabilization not the other way around, in any case Stab() uses DepanEstimate for the vectors. What are you using that uses MAnalyse mv?

[tormento]

in any case Stab() uses DepanEstimate for the vectors. What are you using that uses MAnalyse mv?
Nothing, I think :)

I did some real world benchmarks.

SMDegrain-3.1.2·111~realfinder.avsi with script:
SetMemoryMax()
SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\2_00 13 assassini\assassini.dgi",ct=140,cb=140,cl=0,cr=0)
CompTest24(1)
SMDegrain (tr=4, thSAD=400, refinemotion=false, n16_out=true, mode=0, contrasharp=false, PreFilter=4, truemotion=true, plane=4, chroma=true)
Prefetch(6)

SMDegrain-3.2.0d~Dogway.avsi with script:
SetMemoryMax()
SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\2_00 13 assassini\assassini.dgi",ct=140,cb=140,cl=0,cr=0)
ConvertBits(16)
CompTest24(1)
SMDegrain (tr=4, thSAD=400, refinemotion=false, contrasharp=false, PreFilter=4, truemotion=true, plane=4, chroma=true)
Prefetch(6)

x264.exe --crf 20 --preset slow --aq-mode 2 --colorprim bt709 --colormatrix bt709 --transfer bt709 --range tv --input-depth 16 --output "F:\In\2_00 13 assassini\assassini_4_400_temp\assassini_4_400_out.h264" "F:\In\2_00 13 assassini\assassini_4_400_temp\assassini_4_400.avs"

realfinder: encoded 3624 frames, 10.39 fps, 2904.75 kb/s, 50.20 MB
Dogway: encoded 3624 frames, 13.36 fps, 2903.99 kb/s, 50.18 MB

x265.exe --crf 22 --preset slow --output-depth 10 --colorprim bt709 --colormatrix bt709 --transfer bt709 --range limited --input-depth 16 --dither --output "F:\In\2_00 13 assassini\assassini_4_400_temp\assassini_4_400_out.hevc" "F:\In\2_00 13 assassini\assassini_4_400_temp\assassini_4_400.avs"

realfinder: encoded 3624 frames in 968.31s (3.74 fps), 1740.06 kb/s, Avg QP:25.89
Dogway: encoded 3624 frames in 986.45s (3.67 fps), 1739.71 kb/s, Avg QP:25.90

The speed increase in x264 encoding is massive. I hope there aren't some tradeoffs.

[Dogway]

Wow 13fps, what CPU do you have? I always use Refinemotion though, would also like to have a variable blocksize option in mvtools like in BM3D.
Normally I see a 4% to 5% speed increase, I swapped masktools2 calls with Expr() based ex_tools calls which run faster for HBD, also replaced removegrain blurs with faster alternatives, and process the motion vectors in 8-bit. And expression optimization in luma_rebuild.

Currently I'm trying to optimize for UHD by serving a half size pelclip to MSuper, for that I'm setting pel=1, it runs but I'm not sure if this is correct code-wise maybe if someone can confirm.
Also found that SimpleResize is faster than bilinear for downscaling, so I might use that but doesn't support HBD though.

[tormento]

Wow 13fps, what CPU do you have?
i7-2600k @ 4.6 GHz

I have found that working with 16bit in AVS+ script and leaving all the dithering to x264 (proper HBD aware build from jpsdr) gives a lot of speed increase and better visual results too.

Unfortunately my CPU hasn't enough cores to use AVS+ script plus x265 efficiently.

Can you confirm me that your revised version of SMDegrain is bitdepth transparent, i.e. the output has the same bitplane of the input at any stage of the avsi?

[Dogway]

Ok, seems that converting vector bitdepth back to input bitdepth was strangely not ideal (defaults back to mode=1). In any case client filters can work with vectors of different bitdepth as declared in 2.7.25 of the changelog and also first line of 2.7.19.22.

With the new mvtools2 version DCT is now much faster, so I defaulted DCT to 5, if everyone agrees.

@Tormento, only 8-bit is used for motion vectors, MDegrain uses a MSuper of original bitdepth, and everything else is processed in original bitdepth.

[pinterf]

With the new mvtools2 version DCT is now much faster, so I defaulted DCT to 5, if everyone agrees.

"dct = 5: SATD instead of SAD for luma."
So for this mode you noticed a speed increase? There was a fix indeed, but I couldn't tell the magnitude of this change, did not perform a before-after test.

[tormento]

only 8-bit is used for motion vectors, MDegrain uses a MSuper of original bitdepth, and everything else is processed in original bitdepth.
Even if I feed a HBD source? So some dithering/truncating occurs?

[Dogway]

SMDegrain 3.2.0d final is released. I had changed DCT to 5 by reading the changelog only, now I run some benchs and indeed, no speed gain so reverted. Also optimized ex_luma_rebuild() and some overall cleanup. In the future I want to add some frequency restoration system, this is something that I also looked into 6 years ago.

OLD DCT = 0 7.600fps
OLD DCT = 5 6.300fps
NEW DCT = 5 6.300fps
NEW DCT = 0 7.700fps

@Tormento, yes, the performance/gain ratio is very bad for HBD motion search. ex_luma_rebuild() makes up for an extra 16% precision by converting to PC levels which is very fast with the latest update. You can run tests to see the difference, check below example.

@pinterf, I was confused about bitdepth handling with motion vectors, is the next valid code or do I need MScaleVect()?

super_search = MSuper(rfilter=2)

bv3 = super_search.MAnalyse(isb = true, delta = 3, overlap= 4)
bv2 = super_search.MAnalyse(isb = true, delta = 2, overlap= 4)
bv1 = super_search.MAnalyse(isb = true, delta = 1, overlap= 4)
fv1 = super_search.MAnalyse(isb = false, delta = 1, overlap= 4)
fv2 = super_search.MAnalyse(isb = false, delta = 2, overlap= 4)
fv3 = super_search.MAnalyse(isb = false, delta = 3, overlap= 4)

convertbits(16)
MDegrain3(MSuper(levels=1), bv1, fv1, bv2, fv2 ,bv3, fv3, thSAD=300, thSADC=150)

[pinterf]

SMDegrain 3.2.0d final is released. I had changed DCT to 5 by reading the changelog only, now I run some benchs and indeed, no speed gain so reverted.
dct>0 can be quicker indeed, except for 8x8 block sizes at 8 bit videos. Other than that special format (which is computed by integer dct code) the others are using fftw3, which remained unchanged but the epilog code got a fix in the SSE2 code.

[pinterf]

SMDegrain 3.2.0d final is released. I had changed DCT to 5 by reading the changelog only, now I run some benchs and indeed, no speed gain so reverted. Also optimized ex_luma_rebuild() and some overall cleanup. In the future I want to add some frequency restoration system, this is something that I also looked into 6 years ago.

OLD DCT = 0 7.600fps
OLD DCT = 5 6.300fps
NEW DCT = 5 6.300fps
NEW DCT = 0 7.700fps

@Tormento, yes, the performance/gain ratio is very bad for HBD motion search. ex_luma_rebuild() makes up for an extra 16% precision by converting to PC levels which is very fast with the latest update. You can run tests to see the difference, check below example.

@pinterf, I was confused about bitdepth handling with motion vectors, is the next valid code or do I need MScaleVect()?

super_search = MSuper(rfilter=2)

bv3 = super_search.MAnalyse(isb = true, delta = 3, overlap= 4)
bv2 = super_search.MAnalyse(isb = true, delta = 2, overlap= 4)
bv1 = super_search.MAnalyse(isb = true, delta = 1, overlap= 4)
fv1 = super_search.MAnalyse(isb = false, delta = 1, overlap= 4)
fv2 = super_search.MAnalyse(isb = false, delta = 2, overlap= 4)
fv3 = super_search.MAnalyse(isb = false, delta = 3, overlap= 4)

convertbits(16)
MDegrain3(MSuper(levels=1), bv1, fv1, bv2, fv2 ,bv3, fv3, thSAD=300, thSADC=150)
The code is valid.
The super clip provided to MDegrain must be originated from the same format as the video to denoise.
It can even be a 4:4:4 32 bit float as well, while the vectors can be computed from a plain 8 bit YV12.

I still do not understand what is luma_rebuild for.
"makes up for an extra 16% precision". If this extra range would help with anything then using a 16 bit clip for motion vector search would yield an extra 25600% precision increase. In real life: when you compare the vectors from a 8 bit clip or its ConvertBits(16) counterpart, there will be basically no difference.
What I can imagine that you stretch the Y range and thus it will have a relatively larger weight when chroma is involved.
Btw. you can adjust luma/chroma weights in SAD search with MAnalyze "scaleCSAD" parameter which fine tunes chroma SAD weight relative to luma SAD.

[Dogway]

Bitdepth has more precision, but is not feature bound. Take for example the "opposite" of a TV->PC conversion, a source in log space (or a very low contrast processed source), the SAD will be misleading even in float precision because it doesn't represent human vision.

luma_rebuild() also has a brightening option for the dark areas for the same reason. Human vision is very adaptive to dark (more so in dim/dark surround) so it makes sense to brighten this up to give it more SAD weight and avoid smearing. If this wasn't true we would just try to find motion vectors in linear light and not gamma encoded sources.

I'm using scaleCSAD (MAnalyze), but only for chroma subsampling as written in the docs. Does it have an effect on thSADC of MDegrain? Meaning are they mutually exclusive?

[pinterf]

I see. So it is for lifting the shadows. thsadc can be used independendly from scaleCSAD

[tormento]

yes, the performance/gain ratio is very bad for HBD motion search
I am a bit confused now about the differences between the parameters in your SMDegrain and real.finder's one.

I am looking at your html file, that perhaps you have not updated yet looking at the changelog, and I can see the lsb_in and lsb_out parameters. Weren't they the parameters used when AVS did not support HBD?

Let's have some example:


I have a 8 bit input, I want to internally work in 8 bit and output 8 bit
I have a 8 bit input, I want to internally work in 16/32 bit and output 8/10/12/16 bit
I have a 10/12/16 bit input, I want to internally work in 10/16/32 bit and output 8/10/12/16 bit

Could you please write the correct parameters to be used for each case?

Thanks in advance.

[Dogway]

The documentation hasn't been updated, not even real.finder updated it so it is very old by current terms. I will update when I finish Transforms Pack which will be very soon.

My updated SMDegrain tries to be as simple as possible so I removed redundant arguments like HBD in/out and things like that.
For example 2 precede it with ConvertBits(bits), others need nothing.

[Boulder]

luma_rebuild() also has a brightening option for the dark areas for the same reason. Human vision is very adaptive to dark (more so in dim/dark surround) so it makes sense to brighten this up to give it more SAD weight and avoid smearing. If this wasn't true we would just try to find motion vectors in linear light and not gamma encoded sources.

I've always wondered a bit about luma_rebuild. Using s0=1.0 makes the image generally much darker compared to the original clip than cretindesalpes's original version which has s0=2.0. I thought it would make the clip have less dark areas.

[Dogway]

I don't remember what was the default before my recent edits but I usually try to leave defaults in "safe" settings otherwise people may complain too much, but if there's a consensus I can set it to 2.0, it's still safe (I've gone to 6.0 on a recent project). s0=1.0 by default just converts to PC levels so it appears darker when seen on TV levels preview (ie. AvsPmod)

[tormento]

ExTools
I just updated to 1.7 version and I get:

Script error: Invalid arguments to function 'ex_Yexpr'.
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.2.0d~Dogway.avsi, line 618)
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.2.0d~Dogway.avsi, line 161)
(F:\In\1_30 Febbre da cavallo\febbre_4.avs, line 20)
(F:\In\1_30 Febbre da cavallo\febbre_4_temp\febbre_4_source.avs, line 1)

My script is simple:

SetMemoryMax()
SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\1_30 Febbre da cavallo\febbre.dgi",ct=24,cb=24,cl=0,cr=0)
ConvertBits(16)
SMDegrain (tr=4, thSAD=400, refinemotion=true, contrasharp=false, PreFilter=4, truemotion=true, plane=4, chroma=true)
Prefetch(6)

1.6 is working fine, SMDegrain is the last one.

[Dogway]

Yes, ExTools is a bit developed ahead as I'm making time to solve this (https://forum.doom9.org/showthread.php?p=1945170#post1945170)issue and rebase the other filters.

For the moment use v1.6. I think I will doing other things until the issue which is critical is solved.

[tormento]

For the moment use v1.6. I think I will doing other things until the issue which is critical is solved.
Do you plan to abandon dependencies on Zs_RF_Shared?

[kedautinh12]

Do you plan to abandon dependencies on Zs_RF_Shared?

SMDegrain of Dogway don't dependencies on Zs_RF_Shared

[tormento]

SMDegrain of Dogway don't dependencies on Zs_RF_Shared
Strange. If I recall well, I tried on one of the first beta and it needed.

I will try again with latest one.

[real.finder]

it's still needed for https://github.com/Dogway/Avisynth-Scripts/blob/4f7dcd504721c3a91d0e7cc58cc941faaa9f5d7c/SMDegrain%20v.3.2.0d/SMDegrain%20v3.2.0d.avsi#L39

edit: and also https://github.com/Dogway/Avisynth-Scripts/blob/4f7dcd504721c3a91d0e7cc58cc941faaa9f5d7c/SMDegrain%20v.3.2.0d/SMDegrain%20v3.2.0d.avsi#L145

also there are bug that case error in this line https://github.com/Dogway/Avisynth-Scripts/blob/4f7dcd504721c3a91d0e7cc58cc941faaa9f5d7c/SMDegrain%20v.3.2.0d/SMDegrain%20v3.2.0d.avsi#L170 with subpixel=3 that fixed in https://github.com/Dogway/Avisynth-Scripts/pull/11

[Dogway]

also there are bug that case error in this line https://github.com/Dogway/Avisynth-Scripts/blob/4f7dcd504721c3a91d0e7cc58cc941faaa9f5d7c/SMDegrain%20v.3.2.0d/SMDegrain%20v3.2.0d.avsi#L170 with subpixel=3 that fixed in https://github.com/Dogway/Avisynth-Scripts/pull/11

You mean ex_luma_rebuild()? Expr() has issues dealing with clip luma range as explained in this post (https://forum.doom9.org/showthread.php?p=1945170#post1945170). So development is pretty much halted until solved.

[real.finder]

You mean ex_luma_rebuild()?

it's another one with subpixel=3

[tormento]

Tried pure speed comparison between SMDegrain and NotSMDegrain on an old movie. Latest versions of AviSynth+, AVSI and dll filters.

Script for SMDegrain

SetMemoryMax()
SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\1_30 Febbre da cavallo\febbre.dgi",ct=24,cb=24,cl=0,cr=0)
ConvertBits(16)
SMDegrain (tr=6, thSAD=600, refinemotion=true, contrasharp=false, PreFilter=4, truemotion=true, plane=4, chroma=true)
Prefetch(6)

Script for NotSMDegrain

SetMemoryMax()
SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\1_30 Febbre da cavallo\febbre.dgi",ct=24,cb=24,cl=0,cr=0)
NotSMDegrain (tr=6, thSAD=600, refinemotion=true, n16_out=true, mode=0, contrasharp=false, PreFilter=4, truemotion=true, plane=4, chroma=true)
Prefetch(6)

SMDegrain (tr=6, thSAD=600, refinemotion=true, contrasharp=false, PreFilter=4, truemotion=true, plane=4, chroma=true)

AVSMeter 3.0.8.0 (x64), (c) Groucho2004, 2012-2021
AviSynth+ 3.7.1 (r3413, master, x86_64) (3.7.1.0)

Number of frames: 2712
Length (hh:mm:ss.ms): 00:01:48.480
Frame width: 1920
Frame height: 1032
Framerate: 25.000 (25/1)
Colorspace: YUV420P16

Frames processed: 2712 (0 - 2711)
FPS (min | max | average): 0.918 | 959.8 | 5.764
Process memory usage (max): 2162 MiB
Thread count: 75
CPU usage (average): 72.4%

Time (elapsed): 00:07:50.498

NotSMDegrain (tr=6, thSAD=600, refinemotion=true, n16_out=true, mode=0, contrasharp=false, PreFilter=4, truemotion=true, plane=4, chroma=true)

Frames processed: 2712 (0 - 2711)
FPS (min | max | average): 0.956 | 125000 | 6.124
Process memory usage (max): 1961 MiB
Thread count: 76
CPU usage (average): 73.0%

Time (elapsed): 00:07:22.854

SMDegrain (tr=7, thSAD=700, refinemotion=true, contrasharp=false, PreFilter=4, truemotion=true, plane=4, chroma=true)

Frames processed: 2712 (0 - 2711)
FPS (min | max | average): 0.788 | 1055 | 4.498
Process memory usage (max): 2020 MiB
Thread count: 75
CPU usage (average): 73.2%

NotSMDegrain (tr=7, thSAD=700, refinemotion=true, n16_out=true, mode=0, contrasharp=false, PreFilter=4, truemotion=true, plane=4, chroma=true)

Frames processed: 2712 (0 - 2711)
FPS (min | max | average): 0.477 | 243903 | 5.029
Process memory usage (max): 1783 MiB
Thread count: 74
CPU usage (average): 72.5%

SMDegrain (tr=9, thSAD=900, refinemotion=true, contrasharp=false, PreFilter=4, truemotion=true, plane=4, chroma=true)

Frames processed: 2712 (0 - 2711)
FPS (min | max | average): 0.683 | 714286 | 3.412
Process memory usage (max): 2278 MiB
Thread count: 75
CPU usage (average): 73.1%

NotSMDegrain (tr=9, thSAD=900, refinemotion=true, n16_out=true, mode=0, contrasharp=false, PreFilter=4, truemotion=true, plane=4, chroma=true)

Frames processed: 2712 (0 - 2711)
FPS (min | max | average): 0.835 | 625000 | 3.909
Process memory usage (max): 1964 MiB
Thread count: 75
CPU usage (average): 72.7%

[Dogway]

A little bit slower you mean. I asked last week so now Str is 2.0 by default, that means it does a slight shadow lift, set to 1.0 to disable (previous behaviour). Also for comparisons it's better to limit to 6 frames delta, 7 uses MDegrainN which is slower.

The other edit I did was revert ex_logic to mt_logic (in Contrasharp function), I found that "min" and "max" modes run slower on Expr(), but it might not be the case, test this if the above doesn't solve it.

[tormento]

Also for comparisons it's better to limit to 6 frames delta, 7 uses MDegrainN which is slower.
I compared apples to apples. :) Sometimes a higher delta is required on very grainy sources. Long time ago I asked Pinterf to optimize MDegrain up to 9 but it's not a priority.

I did not use Contrasharpening at all.

So, the difference is because of Str=2.0?

[Dogway]

What I meant is that comparing with high delta you are mostly comparing MAnalyse and little else, to easily check the impact of other things (prefilter, sharpening, etc) using a lower delta is preferred. But this is a debugging tip, I also run real case scripts often to compare.

Yes, I can only think of Str. Another thing to compare is quality. The source seems to be 8-bit, aside of quality gain from lifting the shadows in my version prefiltering is performed in 16-bit (knlmeans + luma_rebuild). I would try to compare dark scenes or with low contrast.

[tormento]

Yes, I can only think of Str. Another thing to compare is quality. The source seems to be 8-bit, aside of quality gain from lifting the shadows in my version prefiltering is performed in 16-bit (knlmeans + luma_rebuild). I would try to compare dark scenes or with low contrast.
Thanks for your clarification.

[Dogway]

Quick heads up. Big overhaul day with latest ExTools changes. Most of the filters, except for Transforms Pack have been updated so give it a look if you wish (and report bugs).
Benchmarks for LSFmod (5 fps performance increase) and GrainFactory3mod (+3 fps) have been updated. Cheers.

[kedautinh12]

Thank Dogway

[wonkey_monkey]

Hi Dogway,

Just copying/expanding on this from the Avisynth+ thread.

You can get a speed increase, at least 20%, from ex_boxblur (should work with the rest as well) with the following changes:


for (px = -krnlrd, krnlrd, 1) {
krnh = px == -krnlrd ? Format("x[{px},0] ") : Format(krnh + "x[{px},0] + ")
}

krnh = krnh + Format(" {wgh} * ")

...

for (py = -krnlrv, krnlrv, 1) {
krnv = py == -krnlrv ? Format("x[0,{py}] ") : Format(krnv + "x[0,{py}] + ")
}

krnv = krnv + Format(" {wgh} * ")

...

str = "x[-1,1] x[0,1] + x[1,1] + x[-1,0] + x[0,0] + " \
+"x[1,0] + x[-1,-1] + x[0,-1] + x[1,-1] + 0.111111 * "



This replaces all the individual pixel multiplications with a single trailing multiplications.

It should also minimise rounding error. Personally I'd add a few more digits to 0.111111 as well (9 ones gives you the minimum rounding error when storing 1/9 as a float)

[Dogway]

Yes, I did the edits and get some 4% performance increase for ex_boxblur(1). Following with radius of 4.


# 294fps 286fps (pp sum, pp mult)
#ex_boxblur(4)

# 298fps 303fps (trailed sum, global div)
#ex_boxblur(4)

# 265fps 268fps (trailed sum, pp mult)
#ex_boxblur(4)

# 314fps 313fps (trailed sum, global mult)
ex_boxblur(4)

Yes, with trailed sum global mult I gain 4%, so 91% of removegrain(19,-1) (483fps) vs ex_boxblur(1, UV=1) (441fps)


It should also minimise rounding error. Personally I'd add a few more digits to 0.111111 as well (9 ones gives you the minimum rounding error when storing 1/9 as a float)
I thought Expr() would round to 6 decimals.


The edited function:
function ex_boxblur(clip a, int "radius", int "radiusV", int "UV", bool "fulls") {

rgb = isRGB(a)
isy = isy(a)
bi = BitsPerComponent(a)

rd = Default(radius, 1) # from 0 to inf
rv = Default(radiusV, rd) # from 0 to inf
UV = Default(UV, rgb ? 3 : 1)
fs = Default(fulls, false)
rd = max(rd, 1)
rv = max(rv, 1)

krnlsz = 2 * rd + 1
krnlrd = krnlsz/2

krnh = ""
for (px = -krnlrd, krnlrd, 1) {
krnh = Format(krnh + "x[{px},0] ")
plsh = px == -krnlrd ? "" : plsh + "+ "
}

krnlsz = 2 * rv + 1
krnlrv = krnlsz/2

krnv = ""
for (py = -krnlrv, krnlrv, 1) {
krnv = Format(krnv + "x[0,{py}] ")
plsv = py == -krnlrv ? "" : plsv + "+ "
}

str = "x[-1,1] x[0,1] x[1,1] x[-1,0] x[0,0] x[1,0] x[-1,-1] x[0,-1] x[1,-1] + + + + + + + + 0.111111111 *"

fbox = rd == 1 && rv == 1
strv = fbox ? str : krnv + plsv + string(1. / krnlrv*2+1) + " *"
strh = krnh + plsh + string(1. / krnlrd*2+1) + " *"

rv == 0 ? a : \
isy ? Expr(a, strv ) : \
UV == 1 ? Expr(a, strv, "" ) : \
Expr(a, strv, ex_UVexpr(strv, UV, bi, rgb, fs), scale_inputs="none")
rd == 0 || fbox ? last : \
isy ? Expr(last, strh ) : \
UV == 1 ? Expr(last, strh, "" ) : \
Expr(last, strh, ex_UVexpr(strh, UV, bi, rgb, fs), scale_inputs="none") }

[wonkey_monkey]

I thought Expr() would round to 6 decimals.

6-7 significant decimal digits is often quoted as the limit for floats, but it's only an approximation. Expr itself reads floats using a stringstream, which takes in all the digits and does its best to convert them to a float.

With 6 ones, the value stored is (approximately):

0.11111100018024444580078125

With 9 or more ones, it's:

0.11111111193895339965820312

[Dogway]

Thanks for the help, I will update and rerun benchmarks to see what I get and update OP if necessary.

[wonkey_monkey]

It occurs to me that a box blur is one case where a non-SIMD variation of Expr could, in theory, work better. The ideal algorithm reads every pixel exactly twice - once when adding it to an accumulator, once when subtracting it. That will give you a near constant-time box blur. Expr has to read the full box of pixels every time.

On another topic, doesn't RemoveGrain(19) only averages the eight neighbours, missing out the central pixel? You should give yourself an extra 11.1% for that ;)

[Dogway]

I did some optimizations to the kernels aside from the above tips and did some benchs and wow, ex_blur(1) is now 96% of removegrain(12)! ex_boxblur(1) is 91% (read below).
Also shaved 4% for ex_xxflate(). Optimized also some convolutions from ex_edge() so they should run faster, for some reason "hprewitt" is bugged I would swear it was working fine before. Also added Overlay_MTools() to the list.

@wonkey_monkey: I don't remember fine but I think I didn't like what it was doing with bigger radius. In any case I removed the central pixel to match removegrain(19) (for parity) and then keep as previous for bigger radius. Now the speed is 97% that of removegrain!

[kedautinh12]

I think can apply above code in here
https://github.com/Dogway/Avisynth-Scripts/blob/9d2f113742a5a7fea3cb4945f78768132e1f69b3/MIX%20mods/FrameRateConverter.avsi#L246

[wonkey_monkey]

I saw BicubicResize on that line of code. Did you know that, with default parameters, BicubicResize is actually a little bit blurry, even if you do a "null" resize with just a infinitessimal shift of pixels? Setting b = 0, c = 0.5 fixes it and is more mathematically correct.

[kedautinh12]

Thanks

[kedautinh12]

About blur(.6)
https://forum.doom9.org/showthread.php?p=1946677#post1946677

[Arx1meD]

Concerning the ex_merge function.
The formula (x*(range_max-z)+y*z)/255 can be simplified to x-(x-y)*(z/range_max)
This will increase the speed.

change
str = "x range_max z - * y z * + range_max /"
to
str = "x x y - z range_max / * -"

[Dogway]

Wow thank you, I was giving it a thought yesterday but couldn't come with any idea, except for the usual reciprocal.

"x x y - z 1 range_max / * * -"

I updated ExTools with several more updates, LSFmod and FrameRateConverter.

[kedautinh12]

Thanks

[kedautinh12]

But how wonkey_monkey said about BicubicResize in Framerateconverter???

[Dogway]

I checked it yesterday and didn't find an issue. I tested null resize with both default b and c, with 0 and 0.5, and raw source and got no differences.

[wonkey_monkey]

I checked it yesterday and didn't find an issue. I tested null resize with both default b and c, with 0 and 0.5, and raw source and got no differences.

BicubicResize optimises out (incorrectly, in my opinion) null resizes. Non-null resizes will reveal the difference. It's very slight, but the defaults do blur.

[Dogway]

I still don't understand, yes, adding a minimal shift shows a difference, it's blurrier:
bicubicresize(1920, 1036, src_left=0.00001, src_top = 0.00001)

If I add catrom coeffs as you suggest it's no blurrier anymore:
bicubicresize(1920, 1036, 0, 0.5, src_left=0.00001, src_top = 0.00001)

But that matches a simple null resize with default values:
bicubicresize(1920, 1036)

[StainlessS]

Wonkey,
Didn't you post some example which clearly showed the problem and which stood out like "a sore thumb".
.

[real.finder]

what about z_BicubicResize ?

[wonkey_monkey]

But that matches a simple null resize with default values:
bicubicresize(1920, 1036)

Because Avisynth deliberately does nothing when it recognises a null resize. What it should do is produce a slightly blurred clip, for consistency. This code makes the problem obvious by using ridiculous b and c parameters:

version
animate(0,16, "bicubicresize", width,height,4,4,-16.0,-16.0, width,height,4,4,16.0,16.0)

You'll see that the middle frame is not consistent with the rest.

Anyway I really just wanted to make sure you were aware that by using BicubicResize with the default parameters you will (unless your resize happens to be a null one) induce a slight, possibly unwanted, blur.

[Dogway]

Continuing from this post (https://forum.doom9.org/showthread.php?p=1947045#post1947045):

@wonkey_monkey: So this is my last attempt at the ternary method. What is wrong about it? The ternary chooses either one or the other, A or C based on the previous comparison which I stored on Q, so I have still 3 elements in the second rightmost comparison.

+"A C < Q@ A AA@ C CC@ ? Q C CC@ A AA@ ? + Z^ "
+"B D < Q@ B BB@ D DD@ ? Q D DD@ B BB@ ? + Z^ "
+"E G < Q@ E EE@ G GG@ ? Q G GG@ E EE@ ? + Z^ "
+"F H < Q@ F FF@ H HH@ ? Q H HH@ F FF@ ? + Z^ "
+"AA EE < Q@ AA A@ EE E@ ? Q EE E@ AA A@ ? + Z^ "
+"BB FF < Q@ BB B@ FF F@ ? Q FF F@ BB B@ ? + Z^ "
+"CC GG < Q@ CC C@ GG G@ ? Q GG G@ CC C@ ? + Z^ "
+"DD HH < Q@ DD D@ HH H@ ? Q HH H@ DD D@ ? + Z^ "
+"A B < Q@ A AA@ B BB@ ? Q B BB@ A AA@ ? + Z^ "
+"C D < Q@ C CC@ D DD@ ? Q D DD@ C CC@ ? + Z^ "
+"E F < Q@ E EE@ F FF@ ? Q F FF@ E EE@ ? + Z^ "
+"G H < Q@ G GG@ H HH@ ? Q H HH@ G GG@ ? + Z^ "
+"CC EE < Q@ CC CC@ EE E@ ? Q EE E@ CC C@ ? + Z^ "
+"DD FF < Q@ DD D@ FF F@ ? Q FF F@ DD D@ ? + Z^ "
+"BB E < Q@ BB B@ E EE@ ? Q E EE@ BB B@ ? + Z^ "
+"D GG < Q@ D DD@ GG G@ ? Q GG G@ D DD@ ? + Z^ "
+"B C < Q@ B BB@ C CC@ ? Q C CC@ B BB@ ? + Z^ "
+"DD EE < Q@ DD D@ EE E@ ? Q EE E@ DD D@ ? + Z^ "
+"F G < Q@ F FF@ G GG@ ? Q G GG@ F FF@ ? + Z^ "

[wonkey_monkey]

A C < Q@ A AA@ C CC@ ? Q C CC@ A AA@ ? + Z^

The problem is that A writes to AA and C writes to CC (twice) regardless of the result of the ternary, because everything is executed. You're throwing away the actual result of both ternaries by adding them together and dumping them in Z.

The stack goes:

A
A, C
(A<C) (followed by a store to Q which does nothing to the stack)
(A<C), A (followed by store to AA which does nothing to the stack)
(A<C), A, C (followed by store to CC which does nothing to the stack)
(A or C) <- result of first "?"
(A or C), Q
(A or C), Q, C (followed by store to CC, does nothing)
(A or C), Q, C, A (followed by store to AA, does nothing)
(A or C), (C or A)


Then you add them together and throw them away into Z.

So that code simplifies to

A @AA C @CC

and that's it.

[Dogway]

As I see it, it was a matter of swap:
A C < Q@ A AA@ C CC@ ? Q C AA@ A CC@ ? +
I just want to tell you that your suggestions in this post (https://forum.doom9.org/showthread.php?p=1946920#post1946920)are flawed, because you can't concatenate two ternaries with no relations between them. You either pop them or do a dumb sum, whichever suits you.

test this and you will get an error:
"A C < A C ? A C < C A ?"

The issue is that poping the stack flushes the previous set variables AA and CC....

EDIT: I saw it.

A C < Q@ A C ? AA@ Q C A ? CC@

[wonkey_monkey]

test this and you will get an error:
"A C < A C ? A C < C A ?"

Because there are two items left on the stack, not one. It wasn't intended to be a complete expression, but an answer to "Is a sort operator possible?"

All suggestions leave the values of both A and C on the stack in order.

[Dogway]

That only answers an ideal test scenario of 2 inputs (no need for a sort algo actually) and not what I asked which was a very well defined sorting network of 8 inputs (https://forum.doom9.org/showthread.php?p=1946915#post1946915).

One output needs to feed the next and so on in order to work. In your example you are not popping anything, if you did the assumed benefit of Q@ (or R@ in your case) wouldn't work.

This works (very slow):

+"A C < A C ? AA^ A C < C A ? CC^ "
+"B D < B D ? BB^ B D < D B ? DD^ "
+"E G < E G ? EE^ E G < G E ? GG^ "
+"F H < F H ? FF^ F H < H F ? HH^ "
+"AA EE < AA EE ? A^ AA EE < EE AA ? E^ "
+"BB FF < BB FF ? B^ BB FF < FF BB ? F^ "
+"CC GG < CC GG ? C^ CC GG < GG CC ? G^ "
+"DD HH < DD HH ? D^ DD HH < HH DD ? H^ "
+"A B < A B ? AA^ A B < B A ? BB^ "
+"C D < C D ? CC^ C D < D C ? DD^ "
+"E F < E F ? EE^ E F < F E ? FF^ "
+"G H < G H ? GG^ G H < H G ? HH^ "
+"CC EE < CC EE ? CC^ CC EE < EE CC ? E^ "
+"DD FF < DD FF ? D^ DD FF < FF DD ? F^ "
+"BB E < BB E ? B^ BB E < E BB ? EE^ "
+"D GG < D GG ? DD^ D GG < GG D ? G^ "
+"B C < B C ? BB^ B C < C B ? CC^ "
+"DD EE < DD EE ? D^ DD EE < EE DD ? E^ "
+"F G < F G ? FF^ F G < G F ? GG^ "


There's no way to reuse the comparison output of R@ or your other optimization suggestions.

[wonkey_monkey]

There's no way to reuse the comparison output of R@ or your other optimization suggestions.

Isn't there? This seems to work:

A C < R@ A C ? AA^ R C A ? CC^

as does this, which duplicates the comparison result without removing it from the stack and is slightly faster:

A C < dup A C ? AA^ C A ? CC^

But anyway, this (which I think was the last of my suggestions, although I may have accidentally deleted my TED Talk) is much faster:

A C min AA^ A C max CC^

And you can squeeze a few more % out with this:

A C dup1 dup1 min AA^ max CC^

[Dogway]

Yes, I don't remember how I tested but it wasn't working at that point. Another option would be to pop out the comparison "A C < R^" but in any case I benchmarked it and barely improved the performance so I went with your min max suggestion. I use dup occasionally but never used dupn, it's not explained in the docs, my guess is it duplicates the n previous element in the stack but this is a guess. Not sure why but it improves performance by 5% so very nice.

[wonkey_monkey]

dup or dup0 copies the top item, dup1 copies the second (zero-indexed) item. It saves Expr from loading the variable from memory again - the value is already in a register because it's on the stack, so it just copies it to another register. "dup0 dup2" would be equally valid in place of "dup1 dup1" because it doesn't matter which order you copy A and C in; I just thought "dup1 dup1" looked nicer, and could be a tiny bit faster (edit: actually it's surprising a lot quicker).

By using swap (which is a free operation) and carefully tracking what's on the stack, you can eliminate the use of variables entirely and get another 4% speedup* (this assumes you're using pinterf's latest test which allows newlines within Expr):


Expr("
x[-1,1] x[1,1] dup1 dup1 min swap2 max
x[0,1] x[-1,0] dup1 dup1 min swap2 max
x[1,0] x[0,-1] dup1 dup1 min swap2 max
x[-1,-1] x[1,-1] dup1 dup1 min swap2 max

swap7 swap1 swap3 dup1 dup1 min swap2 max
swap5 swap1 swap3 dup1 dup1 min swap2 max
swap6 swap1 swap2 dup1 dup1 min swap2 max
swap4 swap1 swap7 dup1 dup1 min swap2 max
swap3 swap1 swap2 dup1 dup1 min swap2 max
swap7 swap1 swap2 dup1 dup1 min swap2 max
swap5 swap1 swap6 dup1 dup1 min swap2 max
swap4 swap1 swap3 dup1 dup1 min swap2 max
swap6 swap1 swap3 dup1 dup1 min swap2 max
swap5 swap1 swap4 dup1 dup1 min swap2 max
swap7 swap1 swap5 dup1 dup1 min swap2 max
swap5 swap1 swap3 dup1 dup1 min swap2 max
swap3 swap1 swap4 dup1 dup1 min swap2 max
swap4 swap1 swap5 dup1 dup1 min swap2 max
swap7 swap1 swap3 dup1 dup1 min swap2 max



swap7 Z^ Z^ Z^ Z^ Z^
swap1
Z^
x swap2
clip
")

Before the line "swap7 Z^ Z^ Z^ Z^ Z^ (where I've left a gap) the stack order is "E HH BB CC D AA FF GG". You can then use swaps and pops to pick out the values you're interested in. In this case, where the final section emulates that median clip thingy, it's BB and GG.

No doubt this will give some people conniptions but there's nothing I can do about that.

A further optimisation would be to delete some of the calculations that don't end up getting used at all, depending on the particular algorithm you're implementing at the time (edit: I orginally wrote 7% speedup above because that was the result of deleting one of the superfluous - for this algorithm - calculations).

[wonkey_monkey]

PS This is about 13x faster (on an AVX2 processor) than you'd get from a non-SIMD C++ plugin - and that's with a fixed kernel size.

[kedautinh12]

Wow, thanks

[Dogway]

dup or dup0 copies the top item, dup1 copies the second (zero-indexed) item. It saves Expr from loading the variable from memory again - the value is already in a register because it's on the stack, so it just copies it to another register. "dup0 dup2" would be equally valid in place of "dup1 dup1" because it doesn't matter which order you copy A and C in; I just thought "dup1 dup1" looked nicer, and could be a tiny bit faster (edit: actually it's surprising a lot quicker).

By using swap (which is a free operation) and carefully tracking what's on the stack, you can eliminate the use of variables entirely and get another 4% speedup* (this assumes you're using pinterf's latest test which allows newlines within Expr):


Expr("
x[-1,1] x[1,1] dup1 dup1 min swap2 max
x[0,1] x[-1,0] dup1 dup1 min swap2 max
x[1,0] x[0,-1] dup1 dup1 min swap2 max
x[-1,-1] x[1,-1] dup1 dup1 min swap2 max

swap7 swap1 swap3 dup1 dup1 min swap2 max
swap5 swap1 swap3 dup1 dup1 min swap2 max
swap6 swap1 swap2 dup1 dup1 min swap2 max
swap4 swap1 swap7 dup1 dup1 min swap2 max
swap3 swap1 swap2 dup1 dup1 min swap2 max
swap7 swap1 swap2 dup1 dup1 min swap2 max
swap5 swap1 swap6 dup1 dup1 min swap2 max
swap4 swap1 swap3 dup1 dup1 min swap2 max
swap6 swap1 swap3 dup1 dup1 min swap2 max
swap5 swap1 swap4 dup1 dup1 min swap2 max
swap7 swap1 swap5 dup1 dup1 min swap2 max
swap5 swap1 swap3 dup1 dup1 min swap2 max
swap3 swap1 swap4 dup1 dup1 min swap2 max
swap4 swap1 swap5 dup1 dup1 min swap2 max
swap7 swap1 swap3 dup1 dup1 min swap2 max



swap7 Z^ Z^ Z^ Z^ Z^
swap1
Z^
x swap2
clip
")

Before the line "swap7 Z^ Z^ Z^ Z^ Z^ (where I've left a gap) the stack order is "E HH BB CC D AA FF GG". You can then use swaps and pops to pick out the values you're interested in. In this case, where the final section emulates that median clip thingy, it's BB and GG.
Wow, I need time to wrap my head around that. Eager to test it out. I had no idea that variables had an overhead, or that dup was any different. I will try to implement this first on the easier modes and check performance. ex_median() is already 3.7.1+ only so formatting is fine.

About your last suggestion, I'm already doing that, removing the last unneeded operations (didn't find much of a benefit though), or for example remove the final division in variances since it doesn't change the comparison output.

I just updated ExTools with new modes. Fixed bokeh which now is a real ring kernel, I tried to create a new mode to sample a bokeh image values, but this would require absolute coordinates, I tried with resizing to kernel size and then stacking (StackVert/Hor limit seems to be 506) but wasn't what I was expecting. Will think about it in the future.

Added removegrain 17 and 18 to ex_median(), weighted median and weighted percentile, median 5x5, and kuwahara mode. I'm also trying to implement Retinex and a corner detection algo.

[wonkey_monkey]

About your last suggestion, I'm already doing that, removing the last unneeded operations (didn't find much of a benefit though)

I was going by the code here (https://forum.doom9.org/showthread.php?p=1947045#post1947045) which has a few redundancies (e.g. the final values of E, D, and FF are not needed). Eliminating just one (I got a bit lost trying to eliminate more) did make a small but noticeable impact on speed.

[Dogway]

Yes, I eliminated those 2 days ago (https://github.com/Dogway/Avisynth-Scripts/blob/29ce2f51307649ee722179ca0aa14846a63bdcbd/ExTools.avsi#L855). I might be able to remove more but I have been more involved in adding new modes and bugfixing. Next versions will be mainly optimizations I guess.

[wonkey_monkey]

By the way how did you come up with that order of min/max swaps in the first place? I can't work out how (for undot2) it determines BB and GG in less than 22 min/max swaps.

[Dogway]

I use sorting networks (http://users.telenet.be/bertdobbelaere/SorterHunter/sorting_networks.html#N8L19D6). Right now I'm trying to port (kinda done), Adaptive Median (http://avisynth.nl/index.php/AdaptiveMedian) by VC Mohan checking the source file, the concept is easy but costly in terms of performance. In any case I'm doing this for the literature so I can have all the algos and how they look in one right place.

[wonkey_monkey]

Hmm. Neat but mindboggling (the sorting networks I mean).

[wonkey_monkey]

Fully optimsed undot2 - 40% faster! :eek:

x[-1,1] x[1,1] dup1 dup1 min swap2 max
x[0,1] x[-1,0] dup1 dup1 min swap2 max
x[1,0] x[0,-1] dup1 dup1 min swap2 max
x[-1,-1] x[1,-1] dup1 dup1 min swap2 max

swap7 swap1 swap3 dup1 dup1 min swap2 max
swap5 swap1 swap3 dup1 dup1 min swap2 max
swap6 swap1 swap2 dup1 dup1 min swap2 max
swap4 swap1 swap7 dup1 dup1 min swap2 max
swap3 swap1 swap2 max
swap6 dup1 dup1 min swap2 max
swap4 swap1 swap5 dup1 dup1 min swap2 max
swap3 swap1 swap2 min
swap4 dup1 dup1 min swap2 max
swap3 swap1 swap2 dup1 dup1 min swap2 max
swap5 swap1 swap3 min
swap2 swap1 swap3 max
swap2 min
swap2 max

x swap2 swap1
clip


This now makes it nearly 18x faster than the equivalent non-SIMD C++ plugin on my computer.

[Dogway]

I tested the undot2 and I got a 10% performance increase, from 317fps (old dup1 dup1 syntax) to 343fps. VERY nice!.

I'm trying to understand the new syntax, I think I got it but got some issues. I wonder why in the first swap3 line you only compute max? Is this to save a comparison check? Following you do a swap6 so you end with A F (I name them to keep track) on the stack and do a min, max when it should be A C.

Stack state after swap3 line:

C F
E G
H D
A B

As an exercise I'm doing the simple undot1:
"x[-1,1] A^ x[0,1] B^ x[1,1] C^ x[-1,0] D^ x[1,0] E^ x[-1,-1] F^ x[0,-1] G^ x[1,-1] H^ "
"x[0,0] A B min C min D min E min F min G min H min A B max C max D max E max F max G max H max clip"
And this was my attempt which doesn't match one to one (and it's slower):
x[0,0] x[-1,1] x[0,1] x[1,1] x[-1,0] x[1,0] x[-1,-1] x[0,-1] x[1,-1]
swap7 swap1 swap6 dup1 dup1 min swap2 max
swap5 dup swap2 min swap5 max
swap1 swap3 dup swap2 max swap4 min
dup1 min swap3 max
dup1 max swap2 min
swap2 dup1 max swap2 min
swap2 dup1 max swap2 min swap clip

EDIT: yikes, I borked it at dup. Edit above but still no match.

[wonkey_monkey]

I'm trying to understand the new syntax, I think I got it but got some issues. I wonder why in the first swap3 line you only compute max? Is this to save a comparison check?

Yes, the result of a min here would never be used so it's preferable not to do it (otherwise it would need to be popped at the end).

Following you do a swap6 so you end with A F (I name them to keep track) on the stack and do a min, max when it should be A C.

Because there's no min calculation, A ends up being removed from the stack entirely. The max changes the stack from "C F E G H D A B" to "C F E G H D B", then on the next line swap6 turns this into "B F E G H D C", ready for C and D to be compared (both min and max, since both are required).

[wonkey_monkey]

Can you give this tool a try?

https://horman.net/expr_sort.php

Select a network from the dropdown, then choose which sorted elements you want (e.g. for undot2, elements 1 and 6), select an ordering for the selected elements to be left on the stack, then hit submit.

Copy the contents of the left box and replace each [#] with an unsorted element (e.g. a pixel reference).

[Dogway]

Woow, so awesome! Saves a lot of time specially for 25 inputs. The problem is except for low inputs it's slower! I do think there's more to it than meets the eye. Following some reasoning.

I think there are some unwritten rules for Expr to optimize performance. I will try list those that come to mind.


Multiplication is faster than division. "x 0.5 *" > "x 2 /"
For better precision calculate the reciprocal within Expr, constants are directly stored so they are reused. "x 1 3 / *" > "x 0.333333333 * "
Ternaries are very slow, try to replace them with min|max when possible. "x y max" > "x y > x y ?"
Dragging stack elements lowers performance. "x 0.5 - x *" > "x x 0.5 - *"
dupn lowers performance for higher "n"
more?


I need to set those into stone to decide what optimization route to go forward with ExTools.
I think the problem with 25 inputs is number 4. swapping might be a free operation but performance suffers because Expr likes continuous operations on the stack. You can see this clearly with the example in my above post, simply swap the location of "x" to the correct stack position and it increases performance by 10%. "x" or "x[0,0]" doesn't make a change (at all).
x[0,-1] x[1,-1]
x[1,0] x[-1,-1]
x[1,1] x[-1,0]
x[-1,1] x[0,1] dup1 dup1 min swap2 max
swap3 dup1 swap2 min swap3 max
swap1 dup1 swap2 max swap2 min
swap3 dup1 swap2 max swap3 min
swap1 dup1 swap2 min swap2 max
swap2 dup1 swap2 min swap2 max
swap2 dup1 swap2 min swap2 max
x swap2 swap1 clip
The degree to which dragging elements behind impacts performance is not yet defined. For example.
old new
"x {th} > 255 scaleb x ?" > "x dup {th} > 255 scaleb swap1 swap2 ?"

old = 471 + 473 + 479 + 474 + 465 + 482 + 481 = 475 mean = 474 median = 1883 (470.75) 50%p
new = 471 + 469 + 486 + 481 + 464 + 468 + 485 = 474.85 mean = 471 median = 1872 (468) 50%p


Another example, each one stacked 3 times to augment difference.
sat=-0.2
Expr("", Format("x A^ 1 {sat} - A * {sat} range_max A - * + ")) # 297 298 297
Expr("", Format("1 {sat} - x dup swap2 * range_max swap1 swap2 - {sat} * +") ) # 294 299 293

As one can see the improvement of not using variables is not clear cut.

So my suggestion for median5 (or long sorting networks) is to bring stack elements to the front from time to time (after 1 or 2 layers).

[wonkey_monkey]

The problem is except for low inputs it's slower!

Slower than what?

[Dogway]

Slower than this (https://github.com/Dogway/Avisynth-Scripts/blob/03ac23f8c7241e390598540038e96871b33e7304/ExTools.avsi#L825).

Tried with this, but didn't help, it needs some manual work.
mode == "median5"? "x[-2,2] x[-1,2] dup1 dup1 min swap2 max
x[0,2] x[1,2] dup1 dup1 min swap2 max
x[2,2] x[-2,1] dup1 dup1 min swap2 max
x[-1,1] x[0,1] dup1 dup1 min swap2 max
x[1,1] x[2,1] dup1 dup1 min swap2 max
x[-2,0] x[-1,0] dup1 dup1 min swap2 max
x[0,0] x[1,0] dup1 dup1 min swap2 max
x[2,0] x[-2,-1] dup1 dup1 min swap2 max
x[-1,-1] x[0,-1] dup1 dup1 min swap2 max
x[1,-1] x[2,-1] dup1 dup1 min swap2 max
x[-2,-2] x[-1,-2] dup1 dup1 min swap2 max
x[0,-2] x[1,-2] dup1 dup1 min swap2 max
swap23 swap1 swap19 dup1 dup1 min swap2 max
swap21 swap1 swap3 dup1 dup1 min swap2 max
swap22 swap1 swap18 dup1 dup1 min swap2 max
swap17 swap1 swap7 dup1 dup1 min swap2 max
swap15 swap1 swap19 dup1 dup1 min swap2 max
swap13 swap1 swap9 dup1 dup1 min swap2 max
swap11 swap1 swap5 dup1 dup1 min swap2 max
swap20 swap1 swap2 dup1 dup1 min swap2 max
swap10 swap1 swap4 dup1 dup1 min swap2 max
swap8 swap1 swap12 dup1 dup1 min swap2 max
swap6 swap1 swap16 dup1 dup1 min swap2 max
swap14 swap1 swap23 dup1 dup1 min swap2 max
swap3 swap1 swap19 dup1 dup1 min swap2 max
swap18 swap1 swap2 dup1 dup1 min swap2 max

N@ P@ D@ O@ U@ E@ S@ L@ M@ R@ V@ J@ K@ X@ F@ Q@ C@ T@ G@ I@ Y@ A@ B@ H@

(...)

[wonkey_monkey]

I wonder if Expr's built-in spilling of values, when they don't fit on the stack, is slower than saving them to variables for some reason.

There is a fault somewhere in yours though - I checked it with random pixel values and it didn't give the correct median.

[Dogway]

Yes I know, I revised it several times but didn't find the culprit. The same with undot1 as stated above (https://forum.doom9.org/showthread.php?p=1947344#post1947344). Might not be my fault who knows.

I think it might also be a matter of dragging behind stack elements as explained above with simple expressions. The following is also very slow, I guess popping them out fixes the issue:
x[-2,2] x[-1,2] dup1 dup1 min swap2 max
x[0,2] x[1,2] dup1 dup1 min swap2 max
x[2,2] x[-2,1] dup1 dup1 min swap2 max
x[-1,1] x[0,1] dup1 dup1 min swap2 max
x[1,1] x[2,1] dup1 dup1 min swap2 max
x[-2,0] x[-1,0] dup1 dup1 min swap2 max
x[0,0] x[1,0] dup1 dup1 min swap2 max
x[2,0] x[-2,-1] dup1 dup1 min swap2 max
x[-1,-1] x[0,-1] dup1 dup1 min swap2 max
x[1,-1] x[2,-1] dup1 dup1 min swap2 max
x[-2,-2] x[-1,-2] dup1 dup1 min swap2 max
x[0,-2] x[1,-2] dup1 dup1 min swap2 max

N^ P^ D^ O^ U^ E^ S^ L^ M^ R^ V^ J^ K^ X^ F^ Q^ C^ T^ G^ I^ Y^ A^ B^ H^
M

EDIT: Above 12 stack elements it starts to bog down.

[wonkey_monkey]

Mine (median5) is faster if you set optSingleMode = true :D Yours does not benefit :(

Unfortunately knowing whether or not to switch it on is probably a black art and may not even be the same from computer to computer. Are you on x86 or x64?

Edit: actually thinking about it, if it's faster on one computer it should be faster on any computer with the same architecture (x86 or x64). It's to do with how many registers are available. Yours gained some speed by keeping more registers free, but lost some due to storing and loading variables. I'm still a little surprised at how close they are though.

When I eventually write my new RPN compiler I plan on including automatic profiling so it should always make the correct choices of such features.

[Dogway]

That one is outdated actually :rolleyes:, I think this might be faster (if we ignore the output difference):
x[-2,2] A^ x[-1,2] B^ x[0,2] C^ x[1,2] D^ x[2,2] E^ x[-2,1] F^ x[-1,1] G^ x[0,1] H^ x[1,1] I^ x[2,1] J^ x[-2,0] K^ x[-1,0] L^ x[0,0] M^
x[1,0] N^ x[2,0] O^ x[-2,-1] P^ x[-1,-1] Q^ x[0,-1] R^ x[1,-1] S^ x[2,-1] T^ x[-2,-2] U^ x[-1,-2] V^ x[0,-2] X^ x[1,-2] Y^ x[2,-2] Z^
A C dup1 dup1 min AA^ max CC^ B I dup1 dup1 min BB^ max II^ D S dup1 dup1 min DD^ max SS^ E R dup1 dup1 min EE^ max RR^ F U dup1 dup1 min FF^ max UU^ G T dup1 dup1 min GG^ max TT^ H J dup1 dup1 min HH^ max JJ^ K L dup1 dup1 min KK^ max LL^ M N dup1 dup1 min MM^ max NN^ O Q dup1 dup1 min OO^ max QQ^ P X dup1 dup1 min PP^ max XX^ V Y dup1 dup1 min VV^ max YY^
AA DD dup1 dup1 min A^ max D^ BB PP dup1 dup1 min B^ max P^ CC SS dup1 dup1 min C^ max S^ EE MM dup1 dup1 min E^ max M^ FF VV dup1 dup1 min F^ max V^ GG KK dup1 dup1 min G^ max K^ HH OO dup1 dup1 min H^ max O^ II XX dup1 dup1 min I^ max X^ JJ QQ dup1 dup1 min J^ max Q^ LL TT dup1 dup1 min L^ max T^ NN RR dup1 dup1 min N^ max R^ UU YY dup1 dup1 min U^ max Y^
A E dup1 dup1 min AA^ max EE^ B H dup1 dup1 min BB^ max HH^ C N dup1 dup1 min CC^ max NN^ D M dup1 dup1 min DD^ max MM^ F G dup1 dup1 min FF^ max GG^ I O dup1 dup1 min II^ max OO^ J P dup1 dup1 min JJ^ max PP^ K V dup1 dup1 min KK^ max VV^ L U dup1 dup1 min LL^ max UU^ Q X dup1 dup1 min QQ^ max XX^ R S dup1 dup1 min RR^ max SS^ T Y dup1 dup1 min TT^ max YY^
AA FF dup1 dup1 min A^ max F^ CC LL dup1 dup1 min C^ max L^ DD GG dup1 dup1 min D^ max G^ EE KK dup1 dup1 min E^ max K^ HH QQ dup1 dup1 min H^ max Q^ II JJ dup1 dup1 min I^ max J^ MM VV dup1 dup1 min M^ max V^ NN TT dup1 dup1 min N^ max T^ OO PP dup1 dup1 min O^ max P^ RR UU dup1 dup1 min R^ max U^ SS YY dup1 dup1 min S^ max Y^
C H dup1 dup1 min CC^ max HH^ G J dup1 dup1 min GG^ max JJ^ I LL dup1 dup1 min II^ max L^ O Z dup1 dup1 min OO^ max ZZ^ S V dup1 dup1 min SS^ max VV^
D II dup1 dup1 min DD^ max I^ HH K dup1 dup1 min H^ max KK^ LL M dup1 dup1 min L^ max MM^ N OO dup1 dup1 min NN^ max O^ P VV dup1 dup1 min PP^ max V^ SS U dup1 dup1 min S^ max UU^ X ZZ dup1 dup1 min XX^ max Z^
E NN dup1 dup1 min EE^ max N^ KK Q dup1 dup1 min K^ max QQ^ L PP dup1 dup1 min LL^ max P^ S Z dup1 dup1 min SS^ max ZZ^ T XX dup1 dup1 min TT^ max X^
BB EE dup1 dup1 min B^ max E^ I LL dup1 dup1 min II^ max L^ JJ TT dup1 dup1 min J^ max T^ N R dup1 dup1 min NN^ max RR^ O SS dup1 dup1 min OO^ max S^ QQ UU dup1 dup1 min Q^ max U^ Y ZZ dup1 dup1 min YY^ max Z^
A B dup1 dup1 min AA^ max BB^ E F dup1 dup1 min EE^ max FF^ GG NN dup1 dup1 min G^ max N^ J OO dup1 dup1 min JJ^ max O^ K RR dup1 dup1 min KK^ max R^ MM Q dup1 dup1 min M^ max QQ^ S T dup1 dup1 min SS^ max TT^ U V dup1 dup1 min UU^ max VV^ X YY dup1 dup1 min XX^ max Y^
CC G dup1 dup1 min C^ max GG^ DD EE dup1 dup1 min D^ max E^ FF N dup1 dup1 min F^ max NN^ H JJ dup1 dup1 min HH^ max J^ M SS dup1 dup1 min MM^ max S^ P R dup1 dup1 min PP^ max RR^ QQ TT dup1 dup1 min Q^ max T^ UU XX dup1 dup1 min U^ max X^ VV Y dup1 dup1 min V^ max YY^
BB C dup1 dup1 min B^ max CC^ F II dup1 dup1 min FF^ max I^ GG HH dup1 dup1 min G^ max H^ J KK dup1 dup1 min JJ^ max K^ L NN dup1 dup1 min LL^ max N^ O PP dup1 dup1 min OO^ max P^ RR U dup1 dup1 min R^ max UU^ V X dup1 dup1 min VV^ max XX^
B D dup1 dup1 min BB^ max DD^ CC E dup1 dup1 min C^ max EE^ FF G dup1 dup1 min F^ max GG^ H LL dup1 dup1 min HH^ max L^ I JJ dup1 dup1 min II^ max J^ K N dup1 dup1 min KK^ max NN^ MM OO dup1 dup1 min M^ max O^ P Q min PP^ R S min RR^
J M max MM^ KK L max LL^ NN O min N^ PP RR min P^ LL MM max M^ N P min NN^ M NN min MM^
MM

I don't mind doing manual profiling this time, well it's what I've doing for 2 months anyway, haha.

[wonkey_monkey]

Yup, faster for me too. What did you change? The ordering of swaps within each layer?

[kedautinh12]

I think need keep output same result cause it will affect whole scripts when just replace some functions. Example: you was replaced blur(.6) with RemoveGrain(1) in Framerateconverter.avsi cause it's only faster but MysteryX said it will make artifact when mask larger

[Dogway]

I just removed unnecessary checks for the last 2 layers. Anyway I will revise it one more time this time with regex so I don't do any mistake.
Today mainly will be profiling and optimizing, if lucky I can release this evening.

@kedautinh12: Yes, these are only drafts. I still haven't updated MIX/EX mods to latest ExTools, it happened that removegrain(12) or blur is not true gaussian but a weighted mean. blur(0.6) was implemented recently as a bare convolution. I also want to optimize Expr with latest tricks but most likely will leave untouched as shown here (https://forum.doom9.org/showthread.php?p=1947419#post1947419).




EDIT: By the way undot was:
x[0,-1] x[1,-1]
x[1,0] x[-1,-1]
x[1,1] x[-1,0]
x[-1,1] x[0,1] dup1 dup1 min swap2 max
swap3 dup swap2 min swap3 max
swap1 dup swap2 max swap2 min
swap3 dup swap2 max swap3 min
swap1 dup swap2 min swap2 max
swap2 dup swap2 min swap2 max
swap2 dup swap2 min swap2 max
x swap2 swap1 clip
It still slower than the old min max succession so I post it here just for reference. I also fixed median5 (an LL^ variable was mistyped) and also weightedp. Overall good optimizations going forward.

[wonkey_monkey]

Can you check if this is faster for median5? It was for me. It uses the second 25-item network instead of the first, which is apparently more Expr-friendly:

Expr(optsinglemode = true, "
x[-2,2] x[-1,2] dup1 dup1 min swap2 max
x[0,2] x[1,2] dup1 dup1 min swap2 max
x[2,2] x[-2,1] dup1 dup1 min swap2 max
x[-1,1] x[0,1] dup1 dup1 min swap2 max
x[1,1] x[2,1] dup1 dup1 min swap2 max
x[-2,0] x[-1,0] dup1 dup1 min swap2 max
x[0,0] x[1,0] dup1 dup1 min swap2 max
x[2,0] x[-2,-1] dup1 dup1 min swap2 max
x[-1,-1] x[0,-1] dup1 dup1 min swap2 max
x[1,-1] x[2,-1] dup1 dup1 min swap2 max
x[-2,-2] x[-1,-2] dup1 dup1 min swap2 max
x[0,-2] x[1,-2] dup1 dup1 min swap2 max
swap23 swap1 swap19 dup1 dup1 min swap2 max
swap21 swap1 swap3 dup1 dup1 min swap2 max
swap22 swap1 swap18 dup1 dup1 min swap2 max
swap17 swap1 swap7 dup1 dup1 min swap2 max
swap15 swap1 swap19 dup1 dup1 min swap2 max
swap13 swap1 swap9 dup1 dup1 min swap2 max
swap11 swap1 swap5 dup1 dup1 min swap2 max
swap20 swap1 swap2 dup1 dup1 min swap2 max
swap10 swap1 swap4 dup1 dup1 min swap2 max
swap8 swap1 swap12 dup1 dup1 min swap2 max
swap6 swap1 swap16 dup1 dup1 min swap2 max
swap14 swap1 swap23 dup1 dup1 min swap2 max
swap3 swap1 swap19 dup1 dup1 min swap2 max
swap18 swap1 swap2 dup1 dup1 min swap2 max
swap7 swap1 swap23 dup1 dup1 min swap2 max
swap21 swap1 swap15 dup1 dup1 min swap2 max
swap9 swap1 swap5 dup1 dup1 min swap2 max
swap4 swap1 swap20 dup1 dup1 min swap2 max
swap12 swap1 swap22 dup1 dup1 min swap2 max
swap11 swap1 swap13 dup1 dup1 min swap2 max
swap16 swap1 swap19 dup1 dup1 min swap2 max
swap8 swap1 swap10 dup1 dup1 min swap2 max
swap14 swap1 swap17 dup1 dup1 min swap2 max
swap6 swap1 swap3 dup1 dup1 min swap2 max
swap2 swap1 swap20 max
swap14 swap1 swap9 dup1 dup1 min swap2 max
swap4 swap1 swap3 dup1 dup1 min swap2 max
swap17 swap1 swap18 dup1 dup1 min swap2 max
swap21 swap1 swap12 dup1 dup1 min swap2 max
swap8 swap1 swap15 dup1 dup1 min swap2 max
swap20 swap1 swap19 dup1 dup1 min swap2 max
swap11 swap1 swap10 dup1 dup1 min swap2 max
swap2 swap1 swap7 dup1 dup1 min swap2 max
swap5 swap1 swap9 dup1 dup1 min swap2 max
swap13 x[2,-2] dup1 dup1 min swap2 max
swap7 swap1 swap20 dup1 dup1 min swap2 max
swap16 swap1 swap22 dup1 dup1 min swap2 max
swap9 swap1 swap11 dup1 dup1 min swap2 max
swap5 swap1 swap20 dup1 dup1 min swap2 max
swap2 swap1 swap21 dup1 dup1 min swap2 max
swap12 swap1 swap6 dup1 dup1 min swap2 max
swap14 swap1 swap7 min
swap22 swap1 swap20 dup1 dup1 min swap2 max
swap18 swap1 swap10 max
swap11 swap1 swap18 dup1 dup1 min swap2 max
swap13 swap1 swap20 dup1 dup1 min swap2 max
swap16 swap1 swap3 dup1 dup1 min swap2 max
swap7 swap1 swap18 dup1 dup1 min swap2 max
swap6 swap1 swap4 dup1 dup1 min swap2 max
swap15 swap1 swap19 dup1 dup1 min swap2 max
swap2 swap1 swap3 max
swap15 swap1 swap7 dup1 dup1 min swap2 max
swap16 swap1 swap18 dup1 dup1 min swap2 max
swap5 swap1 swap9 dup1 dup1 min swap2 max
swap2 swap1 swap4 dup1 dup1 min swap2 max
swap14 swap1 swap20 dup1 dup1 min swap2 max
swap11 swap1 swap7 dup1 dup1 min swap2 max
swap8 swap1 swap15 max
swap16 swap1 swap4 dup1 dup1 min swap2 max
swap11 swap1 swap8 dup1 dup1 min swap2 max
swap2 swap1 swap15 dup1 dup1 min swap2 max
swap12 swap1 swap19 dup1 dup1 min swap2 max
swap3 swap1 swap14 dup1 dup1 min swap2 max
swap6 swap1 swap13 dup1 dup1 min swap2 max
swap7 swap1 swap10 min
swap17 swap1 swap14 max
swap14 swap1 swap6 dup1 dup1 min swap2 max
swap15 swap1 swap12 dup1 dup1 min swap2 max
swap7 swap1 swap17 dup1 dup1 min swap2 max
swap3 swap1 swap9 dup1 dup1 min swap2 max
swap11 swap1 swap8 dup1 dup1 min swap2 max
swap6 swap1 swap10 dup1 dup1 min swap2 max
swap5 swap1 swap16 min
swap3 min
swap10 swap1 swap7 max
swap12 swap1 swap6 dup1 dup1 min swap2 max
swap14 swap1 swap11 max
swap9 swap1 swap3 dup1 dup1 min swap2 max
swap4 swap1 swap12 dup1 dup1 min swap2 max
swap7 swap1 swap3 min
swap4 swap1 swap7 min
swap8 swap1 swap7 max
swap8 swap1 swap9 max
swap9 swap1 swap8 dup1 dup1 min swap2 max
swap5 swap1 swap2 dup1 dup1 min swap2 max
swap8 swap1 swap4 min
swap2 swap1 swap5 min
swap5 swap1 swap7 max
swap3 max
swap5 min
swap3 swap1 swap2 min
swap3 max
swap2 min
min
")

[Dogway]

It runs at 74fps, more or less like my post above, but I went and optimized it further (https://github.com/Dogway/Avisynth-Scripts/blob/791b96fd58522901bb376e4bc36dbdfe694eadd3/ExTools.avsi#L824)and now it runs at 87fps

I already profiled all the modes and should be running at optimal speeds, I'm not sure it's possible to squeeze more out of it. For ex_edge I don't think I will be doing anything, they run pretty fast already.

I tried to do some dup swap thing in some functions like ex_merge, ex_makeadddiff and so, but I don't think they improve performance, if any it kinda feel slower (minimal but still). For ex_binarize smooth mode it was a tie, not sure what to think but left it updated with the new syntax. Tomorrow I will profile GradePack (ex_contrast and ex_levels).


By the way, for anyone reading, rebased MIX/EX mods with latest ExTools.

[wonkey_monkey]

Try this. I got 25% more out of it than my previous one (note that it does NOT use OptSingleMode):

Expr("
x[-2,-2] x[-2,-1] dup1 dup1 min A^ max C^
x[-2,0] x[-2,1] dup1 dup1 min B^ max I^
x[-2,2] x[-1,-2] dup1 dup1 min D^ max S^
x[-1,-1] x[-1,0] dup1 dup1 min E^ max R^
x[-1,1] x[-1,2] dup1 dup1 min F^ max U^
x[0,-2] x[0,-1] dup1 dup1 min G^ max T^
x[0,0] x[0,1] dup1 dup1 min H^ max J^
x[0,2] x[1,-2] dup1 dup1 min K^ max L^
x[1,-1] x[1,0] dup1 dup1 min M^ max N^
x[1,1] x[1,2] dup1 dup1 min O^ max Q^
x[2,-2] x[2,-1] dup1 dup1 min P^ max W^
x[2,0] x[2,1] dup1 dup1 min V^ max X^
A D dup1 dup1 min A^ max D^
B P dup1 dup1 min B^ max P^
C S dup1 dup1 min C^ max S^
E M dup1 dup1 min E^ max M^
F V dup1 dup1 min F^ max V^
G K dup1 dup1 min G^ max K^
H O dup1 dup1 min H^ max O^
I W dup1 dup1 min I^ max W^
J Q dup1 dup1 min J^ max Q^
L T dup1 dup1 min L^ max T^
N R dup1 dup1 min N^ max R^
U X dup1 dup1 min U^ max X^
A E dup1 dup1 min A^ max E^
B H dup1 dup1 min B^ max H^
C N dup1 dup1 min C^ max N^
D M dup1 dup1 min D^ max M^
F G dup1 dup1 min F^ max G^
I O dup1 dup1 min I^ max O^
J P dup1 dup1 min J^ max P^
K V dup1 dup1 min K^ max V^
L U dup1 dup1 min L^ max U^
Q W dup1 dup1 min Q^ max W^
R S dup1 dup1 min R^ max S^
T X dup1 dup1 min T^ max X^
A F max F^
C L dup1 dup1 min C^ max L^
D G dup1 dup1 min D^ max G^
E K dup1 dup1 min E^ max K^
H Q dup1 dup1 min H^ max Q^
I J dup1 dup1 min I^ max J^
M V dup1 dup1 min M^ max V^
N T dup1 dup1 min N^ max T^
O P dup1 dup1 min O^ max P^
R U dup1 dup1 min R^ max U^
S X dup1 dup1 min S^ max X^
C H dup1 dup1 min C^ max H^
G J dup1 dup1 min G^ max J^
I L dup1 dup1 min I^ max L^
O x[2,2] dup1 dup1 min O^ max Y^
S V dup1 dup1 min S^ max V^
D I max I^
H K dup1 dup1 min H^ max K^
L M dup1 dup1 min L^ max M^
N O dup1 dup1 min N^ max O^
P V dup1 dup1 min P^ max V^
S U dup1 dup1 min S^ max U^
W Y dup1 dup1 min W^ max Y^
E N dup1 dup1 min E^ max N^
K Q dup1 dup1 min K^ max Q^
L P dup1 dup1 min L^ max P^
S Y dup1 dup1 min S^ max Y^
T W dup1 dup1 min T^ max W^
B E max E^
I L dup1 dup1 min I^ max L^
J T dup1 dup1 min J^ max T^
N R dup1 dup1 min N^ max R^
O S dup1 dup1 min O^ max S^
Q U dup1 dup1 min Q^ max U^
X Y min X^
E F max F^
G N dup1 dup1 min G^ max N^
J O dup1 dup1 min J^ max O^
K R dup1 dup1 min K^ max R^
M Q dup1 dup1 min M^ max Q^
S T dup1 dup1 min S^ max T^
U V min U^
W X min W^
C G max G^
F N dup1 dup1 min F^ max N^
H J dup1 dup1 min H^ max J^
M S dup1 dup1 min M^ max S^
P R dup1 dup1 min P^ max R^
Q T min Q^
U W min U^
F I max I^
G H max H^
J K dup1 dup1 min J^ max K^
L N dup1 dup1 min L^ max N^
O P dup1 dup1 min O^ max P^
R U min R^
H L max L^
I J max J^
K N dup1 dup1 min K^ max N^
M O dup1 dup1 min M^ max O^
P Q min P^
R S min R^
J M max M^
K L max L^
N O min N^
P R min P^
L M max
N P min
min
")

I think there is still some work to be done. I'll update my online tool at some point as it now inserts automatic pixel references (when it can) and also has a variable-ised output option.

[wonkey_monkey]

Made a couple of optimisations to the above post. I won't edit it further to avoid confusion.

[Dogway]

Yes I got like 2 or 3 more fps (82fps) vs mine (79fps, CPU now is in lazy state). It's like an optimized version of the variable syntax.


EDIT: woow so fast (your edit), now 94fps vs 85fps
EDIT2: I mainly use 5x5 sorting networks for median5 and adaptive, but I'm thinking on adding it also to weighted 50% percentile, because it kinda makes more sense there than 3x3, although output wasn't what I expected...

[wonkey_monkey]

I'm realising just now that although these sorting networks are optimised for sorting, they are not necessarily optimised for finding. My generator makes the best optimisation it can out of them (mathematically speaking), but it can lead to non-optimal results. This may only affect odd-numbered networks. I'll continue to investigate.

[Dogway]

Thanks for the effort. Probably there might be finding algos (https://www.geeksforgeeks.org/searching-algorithms/) around but I didn't research that. In any case you can use reverse elimination as we have been doing.

[Dogway]

@wonkey_monkey: I could achieve a 5% improvement in median5 by realizing that my median algo was the same than undot4 (clip to closest). This requires one less input for the sorting network.

I'm also adding vertical median, matching verticalcleaner mode=1.
"x[0,1] x[0,-1] dup1 dup1 max swap2 min x swap2 clip"
Didn't have much luck with mode=2 (source code reads intelligible for me). I tried to apply the description:

Let b1, b2, c, t1, t2 be a vertical sequence of pixels. The center pixel c is to be modified in terms of the 4 neighbours. For simplicity let us assume that b2 <= t1. Then in mode 1, c is clipped with respect to b2 and t1, i.e. c is replaced by max(b2, min(c, t1)). In mode 2 the clipping intervall is widened, i.e. mode 2 is more conservative than mode 1. If b2 > b1 and t1 > t2, then c is replaced by max(b2, min(c, max(t1,d1) )), where d1 = min(b2 + (b2 - b1), t1 + (t1 - t2)). In other words, only if the gradient towards the center is positive on both clipping ends, then the upper clipping bound may be larger. If b2 < b1 and t1 < t2, then c is replaced by max(min(b2, d2), min(c, t1)), where d2 = max(b2 - (b1 - b2), t1 - (t2 - t1)). In other words, only if the gradient towards the center is negative on both clipping ends, then the lower clipping bound may be smaller.

But didn't yield the expected result, so played with the expression a bit and got a close match.

x[0,2] T2^ x[0,1] T1^
x[0,-2] B1^ x[0,-1] B2^
B2 B1 > T1 T2 > & B2 B2 B1 - + T1 T1 T2 - + min T1 ? MA^
B2 B1 < T1 T2 < & B2 B1 B2 - - T1 T2 T1 - - max B2 ? MI^
x B1 MA min x T1 max max T1 MI max x min B2 max clip

Could you take a look at it? not sure where it might be failing. I will also add temporal median.

###################

On another note, I'm also developing a dot crawl dechecker filter. I will make use of your fantastic kernel here (https://forum.doom9.org/showthread.php?p=1751654#post1751654), but that will be the last discretion of three, SAD, spatial match and temporal match.

About SAD I'm not sure how to implement it, per block or per pixel.

Per pixel(?)
a=selectevery(1,0)
b=selectevery(1,-1)
Expr(a,b,"x[1,1] XA^ x[1,0] XB^ x[1,-1] XC^ x[-1,1] XD^ x[-1,0] XE^ x[-1,-1] XF^ x[0,1] XG^ x[0,-1] XH^
y[1,1] YA^ y[1,0] YB^ y[1,-1] YC^ y[-1,1] YD^ y[-1,0] YE^ y[-1,-1] YF^ y[0,1] YG^ y[0,-1] YH^
XA YA - abs XB YB - abs XC YB - abs XD YB - abs XE YE - abs XF YF - abs XG YG - abs XH YH - abs + + + + + + + ","")



EDIT: By the way mt_motion(last, thy1=0,thy2=255,tht=255,U=-128,V=-128) pretty much matches the next
thSAD=300 / 7.111
Expr(a,b,"x[1,1] XA^ x[1,0] XB^ x[1,-1] XC^ x[-1,1] XD^ x[-1,0] XE^ x[-1,-1] XF^ x[0,1] XG^ x[0,-1] XH^
y[1,1] YA^ y[1,0] YB^ y[1,-1] YC^ y[-1,1] YD^ y[-1,0] YE^ y[-1,-1] YF^ y[0,1] YG^ y[0,-1] YH^
XA YA - abs XB YB - abs XC YB - abs XD YB - abs XE YE - abs XF YF - abs XG YG - abs XH YH - abs + + + + + + + 8 / W@ 9 * {thSAD} scalef < 0 W ?","")

[wonkey_monkey]

I think you might have overlooked the part which says "For simplicity let us assume that b2 <= t1" although that whole description is pretty opaque to me.

[Dogway]

Thanks. Found a source I could read.

ExTools updated to v4.0.
-ex_median() new modes; vertical medians (ported from verticalcleaner), temporal medians (medianT and medianT5) and spatio-temporal medians (medianST, ML3D, ML3Dex, and BDM). Some modes overlap to Clense and MedianBlur.
-ex_kawase(), add logarithmic multipass
-ex_repair(), port half the modes from removegrain repair()
-ex_smooth(), 2 small bugfixes

[Dogway]

ExTools updated to v4.1.
-ex_bilateral() new filter. Performs (much) slower than Dither_bilateral16() not sure why since it's just a few operators (Edit: yes, exp operator is uber slow, pow is slower though). Anyhow the output is cleaner and HBD native (and includes a dejaggie arg). Won't make a SmoothGrad filter unless the speed issue is solved.
-ex_median() new mode MMF. Fixed weightedp (for real)

[Dogway]

ExTools updated to v4.2.
-ex_median() new modes; SixNN (6 Nearest-Neighbours), PML (Planar Multi-Level), and removegrain modes 26 and 27. I plan to implement Kalman motion estimation (if possible) to avoid using mvtools.
-ex_bilateral() optimized a bit, still runs at around 93fps mainly due to the exp operator.

I will be uploading to OP a comparison chart for all the modes in a few minutes.

[tormento]

ExTools updated to v4.2.
Would you mind to create a greyscale mode for SMDegrain and, if needed, ExTools?

With old movies, where more intensive filtering is required, I tried some times ago to set plane=0, chroma=false but I had worst results, mostly with higher bitrate when encoding, than leaving on "color mode".

[Boulder]

Greyscale() at the end of your script would do the same if those two parameters already bypass any chroma processing for speedups.

[tormento]

Greyscale() at the end of your script would do the same if those two parameters already bypass any chroma processing for speedups.
Not so easy...

[Dogway]

If the clip doesn't have colors then the UV planes should be mostly blank, so analyzing vectors from them looks like a bad idea.
At least for ExTools you can set UV=1 and chroma planes will be copied, I would need to check out SMDegrain but I think it does the same. I have yet to rework the Contrasharpening area, there are at least three methods for antiringing that I have to study, closely related to limiting so I also need to assess spatial and temporal limiting, it's not my forte so it will take some time.

[real.finder]

If the clip doesn't have colors then the UV planes should be mostly blank, so analyzing vectors from them looks like a bad idea.


yes I told him this back then https://forum.doom9.org/showthread.php?p=1917477#post1917477

[tormento]

yes I told him this back then https://forum.doom9.org/showthread.php?p=1917477#post1917477
But I couldn't find a solution. Have some tries and tell me.

[real.finder]

But I couldn't find a solution. Have some tries and tell me.

remove UV with ConvertToY/ConvertToY8 after source call and you are safe and you should get more speed as well

[Dogway]

If the grain is very high solutions have been given before. You can try with this (https://forum.doom9.org/showthread.php?p=1911107#post1911107). It's the feature I've been long wanting to implement to SMDegrain.

[Dogway]

ExTools updated to v4.3.
-ex_kawase() fixed a stupid mistake and added 2 new modes, fibonacci and tribonacci sequence.
-ex_repair() implemented 4 temporal repair modes, this replaces TemporalRepair from RGTools which by the way, is broken.

GradePack updated to v2.5.
-ex_glow() new filter to add glow, 2 modes fibonacci and tribonacci, it's very slow, is not optimized, only for proof of concept.

I also updated OP with a ranking benchmarks of all the modes (ex_median, blurs and ex_edge). I want to upload also a comparison chart for all blend modes.

[Dogway]

ExTools updated to v4.4.
-ex_smartblur(). Filter for blurring similar to Photoshop's filter. Less gentle blur than ex_bilateral() but much faster.
-ex_median() new mode EMF (Extended Median Filter)
-ex_vinverse() removed the monolithic script and included into ExTools with some additional updates.
-ex_boxblur() updated "weighted" mode weights to use binomial coefficients. Now it should match ex_blur() (but faster) and removegrain(12), blur(), etc

QTGMC updated to v3.5 based on 3.382s
Practically same speed for 8-bits (even with Expr unsupported LUT based calculations) and about 15% to 20% more performant for HBD. See OP for benchmarks.
Changelog:
- Replaced masktools2 with Expr calls via ExTools pack wrappers (except mt_merge)
- Merged redundant expressions
- Replaced blurs and medians with ExTools equivalents
- Replaced Vinverse_avsi() with ex_vinverse() (HBD) and vinverse() (8bit)
- Removed Dither support
- Formatting and small optimizations




I also want to post a list of all the median filters I could find. They happen to be much more than I thought, so I'm not including them all, although people can help out if interested to make a good library, despite some of them protected under a paywall or not possible with Expr expressions (recursions, etc)
From all of them the ones I'm interested most are: IDBA, TSND, RAA, ADKI, and AUTMF.

Convolution algorithms:
# ATMF - Asymmetric Trimmed Median Filter
# SMF - Switching Median Filter
# WMF - Weighted Median Filter
# CWMF - Center Weighted Median Filter
# DWMF - Directional Weighted Median Filter
# MDWMF - Modified Directional Weighted Median Filter
# RWMF - Recursive Weighted Median Filter
# PSMF - Progressive Switching Median Filter
# SMF-BDND - Switching Median Filter with Boundary Discriminative Noise Detection
# BDSMF - Boundary Discriminative Switching Median Filtering
# DBF - Decision Based Filter
# DBA - Decision Based Algorithm Median Filter
# DBA2 - Decision Based Algorithm 2 Median Filter
# MDBUTMF - Modified Decision Based Un-symmetric Trimmed Median Filter
# DBUTMF - Decision Based Un-symmetric Trimmed Median Filter
# ACWMF - Adaptive Center Weighted Median Filter
# NID - New Impulse Detector
# AMF - Adaptive Median Filter
# FUF - Fuzzy Filter
# UF -
# ADKI - Adaptive Decision based Kriging Interpolation
# IMF - Improved adaptive Median Filter ( Ha et al. (2016))
# IAMF - Improved Adaptive Median Filtering ( (Wang and Li, 2010))
# SAMF - Selective Adaptive Median Filter
# MDBSMF - Multiple Decision Based Switching Median Filter
# TSMF/TSF - Tri-State Median Filter
# NTSMF - New Tri-State Switching Median Filter
# NASMF - Noise Adaptive Soft-switching Median Filter
# FRMF - Fuzzy Rule-based Median Filtering
# RAA - Removal of salt and pepper an Adaptive Approach
# IDBA - Improved decision based algorithm
# HFFLND - Hybrid Filter and Fuzzy Logic Noise Detector
# NSPF - New Salt and Pepper Filter. (Biswal and Bohi, 2013)
# FEMF - Fast and Efficient Median Filter
# AMF-LPD - Adaptive Median Filtering on Local Pixel Distribution
# SSMF - Sorted Switching Median Filter
# TSND - Two-Stage Noise Detector
# IEMF - Improved Extremum and Median value Filter
# ADPSMF - Absolute Difference based Progressive Switching Median Filter
# IMFA - Improved Median Filtering Algorithm
# MBDND - Modified Boundary Discriminative Noise Detection
# MSSMF - Multistate Switching Median Filter
# RVINE - Random Valued Impulse Noise Elimination
# RVINENF - Random Valued Impulse Noise Elimination using Neural Filter
# RVINENFF - Random Valued Impulse Noise Elimination using Neuro-Fuzzy Filter
# EPF - Edge Preserving Filter
# NBPPFT - Neural Based Post Processing Filtering Technique
# FTPHR - Filtering Technique for Preserving Homogeneous Region
# FTPHRAER - Filtering Technique for Preserving Homogeneous Region and Edge Region
# FTHCDI - Filtering Technique for Highly Corrupted Digital Images
# CNFT - Combined Neural based Filtering Technique
# FFBPAMNEM1- Feed Forward Back Propagation Algorithm for Multiple Noise Elimination Method 1
# FFBPAMNEM2- Feed Forward Back Propagation Algorithm for Multiple Noise Elimination Method 2
# ANFIS - Adaptive Neuro-fuzzy Inference System
# HNFF - Hybrid Neuro-Fuzzy Filter
# TSHNFF1 - Two Stage Hybrid Neuro-Fuzzy Filter 1
# TSHNFF2 - Two Stage Hybrid Neuro-Fuzzy Filter 2
# MNENFF1 - Multiple Noise Elimination using Neuro-Fuzzy Filter 1
# MNENFF2 - Multiple Noise Elimination using Neuro-Fuzzy Filter 2
# INEDI1 - Intelligent Network for Enhancing Digital Images 1
# INEDI2 - Intelligent Network for Enhancing Digital Images 2

# BF - Bilateral Filter
# FBF - Fuzzy Bilateral Filter
# INR - Impulse Noise Reduction
# FRINR - Fuzzy Random Impulse Noise Reduction
# FRINRM - Fuzzy Random Impulse Noise Reduction Method
# FIDRM - Fuzzy Impulse noise Detection and Reduction Method
# FIDRMC - Fuzzy Impulse noise Detection and Reduction Method for Color Images
# HFRMC - Histogram-based Fuzzy Restoration Method for Color Images
# FISF - Fuzzy Inference System Filter
# AUTMF - Adaptive Unsymmetrical Trim-Based Morphological Filter
# MMEM - Minimum-Maximum Exclusive Mean Filter
# DBA - Decision-Based Asymmetrical trimmed median filter
# FRF - Fuzzy Rank-ordered Filter
# LABSVMF - cieLAB Switching Vector Median Filter
# FCG - Fuzzy Color preserving Gaussian filter

# Non Linear (for RGB images)
# VMF - Vector Median Filter
# TVMF - Trimmed Vector Median Filter
# EVMF - Extended Vector Median Filter
# FMVMF - Fast Modified Vector Median Filter
# FSVF - Fast Similarity Vector Filter
# DVMF - Directional Vector Median Filter
# GVMF - Generalized Vector Median Filter
# RCVMF - Rank Conditioned Vector Median Filter
# RCTVMF - Rank Conditioning and Threshold Vector Median Filter
# CLMMF - Crossing Level Median Mean Filter
# VDF - Vector Directional Filter
# BVDF - Basic Vector Directional Filter
# GVDF - Generalized Vector Directional Filter
# DDF - Directional Distance Filter
# WVMF - Weighted Vector Median Filter
# TWVMF - Trimmed Weighted Vector Median Filter
# EXWVMF - Extended Weighted Vector Median Filter
# ROWVMF - Rank Order Weighted Vector Median Filter
# WVDF - Weighted Vector Directional Filters
# GA WVDF - Genetic Algorithm Weighted Vector Directional Filter
# CWVMF - Center-Weighted Vector Median Filter
# AVMF - Adaptive Vector Median filter
# ABVDF - Adaptive Basic Vector Directional Filter
# MSVMAF - Multiclass Support Vector Machine based Adaptive Filter
# ATCC - Adaptive Threshold and Color Correction
# RSVF/RSVMF- Robust Switching Vector Filter
# AMMF - Adaptive Marginal Median filter
# MSMF - Modified Switching Median Filter
# RSDDF - Robust Switching Directional Distance filter
# ACWVMF - Adaptive Center-Weighted Vector Median Filter
# ARWSF - Adaptive rank weighted switching filter
# PGA - Peer Group Averaging
# PGSF - Peer Group Switching Filter
# MPG - Modified Peer Group
# PGVF - Peer Group Vector Filter
# FPGF - Fast Peer Group Filters
# NPGF - Novel Peer Group Filter
# FWAF - Fuzzy Weighted Average Filter
# FVMF - Fuzzy Vector Median Filter
# FVDF - Fuzzy Vector Directional Filter
# FOVF - Fuzzy Ordered Vector Filter
# FHF - Fuzzy Hybrid Filter
# ANNF - Adaptive Nearest-Neighbor Filter
# ANNMF - Adaptive Nearest-Neighbor Multichannel Filter
# AHMF - Adaptive Hybrid Multichannel Filter
# FIDRM - Fuzzy Impulse Detection and Reduction Method
# VMRHF - Vector Median Rational Hybrid filter
# SAHVF - Structure-Adaptive Hybrid Vector Filter
# SWAV - Structure Weighted Average Filter
# SVMF - Sigma Vector Median Filter
# SBVDF - Sigma Basic Vector Directional Filter
# SDDF - Sigma Directional Distance Filter
# ASVMF - Adaptive Sigma Vector Median Filter
# ASBVDF - Adaptive Sigma Basic Vector Directional Filter
# ASDDF - Adaptive Sigma Directional Distance Filter
# ASVMF - Adaptive Sigma Vector Median Filter
# EVMF - Entropy Vector Median Filter
# QSF - Quaternion Switching Filter
# QVMF - Quaternion Vector Median Filter
# QSVMF - Quaternion Switching Vector Median Filter

[Dogway]

ExTools updated to v4.5.
-ex_median() ported spatio-temporal DeGrainMedian modes
-ex_repair() ported repair(16)

Logo updated to v12.0.

For some reason I couldn't load DeGrainMedian with MP_Pipeline win32 mode so I couldn't double check if the output matched, nor benchmark it against. The filter is slowish but I'm not sure how to optimize it further.
I also wanted to port GrapeSmooth but closed source it seems.

I'm also still porting Adaptive Sharpen, I came to a little halt so may ask later for some help.

[real.finder]

For some reason I couldn't load DeGrainMedian with MP_Pipeline

there are https://github.com/Asd-g/AviSynth-vsDeGrainMedian

[kedautinh12]

I also wanted to port GrapeSmooth but closed source it seems.

I'm also still porting Adaptive Sharpen, I came to a little halt so may ask later for some help.

Ask Asd-g for port from Vapoursynth

[real.finder]

Ask Asd-g for port from Vapoursynth

Adaptive Sharpen vs link? I didn't find it, they say that Adaptive Sharpen is better than CAS

[Dogway]

there are https://github.com/Asd-g/AviSynth-vsDeGrainMedian

woow, great, source is much more readable. Besides I got my algos wrong, I think I borked them at last minute trying to fix a stupid issue before uploading.

I'm porting this (https://forum.doom9.org/showthread.php?t=172131)Adaptive Sharpen. Currently I'm stuck at local minima and maxima.

Since I cannot do recursion in Expr I deployed a small script to determine what the string ends being, but it "crashes" (endless loop?). I'm looking into porting this version first, optimize, then port this (https://gist.github.com/igv/8a77e4eb8276753b54bb94c1c50c317e) further optimized version, and if possible adapt it to turn it into Adaptive Contrasharpen. Also I don't like how they want to sharpen in linear or sigmoid gamma, sharpen should be performed in gamma light.


Reference. (https://github.com/bacondither/Adaptive-sharpen/blob/0da6654e464c4136e3a68d80f5cd3bd7563762b9/shaders/Adaptive-sharpen%20-%20Pass%20two.hlsl#L204)
luma0 = "luma0 " luma12 = "luma12 "
luma1 = "luma1 " luma13 = "luma13 "
luma2 = "luma2 " luma14 = "luma14 "
luma3 = "luma3 " luma15 = "luma15 "
luma4 = "luma4 " luma16 = "luma16 "
luma5 = "luma5 " luma17 = "luma17 "
luma6 = "luma6 " luma18 = "luma18 "
luma7 = "luma7 " luma19 = "luma19 "
luma8 = "luma8 " luma20 = "luma20 "
luma9 = "luma9 " luma21 = "luma21 "
luma10 = "luma10 " luma22 = "luma22 "
luma11 = "luma11 " luma23 = "luma23 "
luma24 = "luma24 "
for (i = 0, 2, 1)
{
temp=""

for (j = 0, 23-i, 2)
{
jp = j+1
temp = Eval(Format("luma{j}"))
Eval(Format(""" luma{j} = + string(luma{j}) + string(luma{jp}) + " min " """))
Eval(Format(""" luma{jp} = + string(temp) + string(luma{jp}) + " max " """))
}


for (jj = 24-i, i+1, -2)
{
temp = Eval(Format("luma{i}"))
Eval(Format(""" luma{i} = + string(luma{i}) + string(luma{jj}) + " min " """))
Eval(Format(""" luma{jj} = + string(temp) + string(luma{jj}) + " max " """))

jl = 24-i
js = jj-1
temp = Eval(Format("luma{jl}"))
Eval(Format(""" luma{jl} = + string(luma{jl}) + string(luma{js}) + " max " """))
Eval(Format(""" luma{js} = + string(temp) + string(luma{js}) + " min " """))
}
}

subtitle(last,luma1,size=10)


CAS is unsharp mask based as many other sharpeners, they mainly differ on how to avoid ringing, Adaptive Sharpen is another sharpening flavour. At some point I will try to compare all of them and do some evaluation. There's also Smart Sharp, qUINT sharp, NLM Sharp, there are some interesting *.mp4 guy sharpeners as well and I don't forget Plum by feisty2.

[kedautinh12]

Adaptive Sharpen vs link? I didn't find it, they say that Adaptive Sharpen is better than CAS

Sr, i think it's Vapoursynth filter

[real.finder]

Dogway, maybe it's better to make it a plugin? you can make it only with C/C++ code and then asked another person to add optimized asm code for speed up if you can't do asm codes

[Dogway]

Updated DeGrainMedian modes. Actually it wasn't my fault, I really have to go through hoops to make the algos work with the variables bugs.

Yes, plugin should be really easy for people into C, mostly 1:1 port. I'm also doing this for learning though (ex mods) and library, while mix mods are speed oriented. I'm not sure if after 6 years nobody tackled Adaptive Sharpen, it might be lack of interest or maybe hard to implement, I don't know. In any case if you want to go HD or UHD sharpening you might need to apply high sigma blurs which are only fast on frequency domain, I doubt scaling down, blur (kawase?) then up is faster than that.

[real.finder]

feisty2 said that new c++ is easy like scripting, maybe you can try use last c++ with GCC and C api of avs/avs+

[Dogway]

Yes, I have that in the back of my mind. GCC is lightweight, anyway I don't think writing asm is easy.
Actually I just wanted to make a few filters ExTools and TransformsPack and get going since I have an VR I bought 2 months ago and still haven't barely opened (need to install Win10/11, blabla, sloth).

[tormento]

remove UV with ConvertToY/ConvertToY8 after source call and you are safe and you should get more speed as well
My script:

SetMemoryMax()
SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\1_44 Harvey\harvey.dgi",ct=0,cb=0,cl=232,cr=232)
ConvertToY()
SMDegrain (tr=4, thSAD=400, trymany=true, refinemotion=true, contrasharp=false, PreFilter=4, truemotion=true, plane=0, chroma=false)
Prefetch(6)

Error:

Script error: syntax error
(D:/Programmi/Media/AviSynth+/plugins64/ExTools-4.5b~Dogway.avsi, line 1246, column 0)
(F:\In\1_44 Harvey\harvey_Y16.avs, line 3)
(F:\In\1_44 Harvey\harvey_Y16_temp\harvey_Y16_source.avs, line 1)

ExTools4.5b is buggish. With 4.4 it works ok.

P.S: How can I get the equivalent of color 16 bit on 4:0:0? 8 bit is enough for greyscale?

[Dogway]

Thanks, updated, it was a stupid typo.

You can improve denoising with HBD.
ConvertToY()
Convertbits(16)

By the way, the above iterator sample was as expected a sorting algo, so good to know another tool for the toolbox. The error dialog was for subtitle(), instead I wrote out the output to a file. Now Adaptive Sharpen is in debug phase.

[Arx1meD]

I also want to post a list of all the median filters I could find. They happen to be much more than I thought, so I'm not including them all, although people can help out if interested to make a good library, despite some of them protected under a paywall or not possible with Expr expressions (recursions, etc)
From all of them the ones I'm interested most are: IDBA, TSND, RAA, ADKI, and AUTMF.

Convolution algorithms:
# RAA - Removal of salt and pepper an Adaptive Approach


There is a ModPlus plugin and it includes a SaltPepper filter. It can be found here (also source code) - ModPlus (http://avisynth.nl/index.php/ModPlus).
I don’t know how useful it will be for you.

[kedautinh12]

ManyPlus have ModPlus with new updated
http://www.avisynth.nl/users/vcmohan/manyPlus/manyPlus.html

[Dogway]

Finished the port of Adaptive Sharpen (https://github.com/Dogway/Avisynth-Scripts/blob/master/EX%20mods/Adaptive_Sharpen.avsi) (phew!)

It's very slow though, I thought that processing in 8-bit would make it somewhat production ready speed, but no, it's slower. Maybe when/if we get LUT calculations in Expr though.
There are already some optimization tricks to make it faster so the next step would only be to port the optimized HLSL from igv (https://gist.github.com/igv/8a77e4eb8276753b54bb94c1c50c317e).
For the time being use it in 8 or 32-bit. Haven't tested in other depths.
Eventually I want to create a sharpeners pack with FineSharp, SeeSaw, etc and try to find a good candidate for Contrasharp replacement in SMDegrain.


I checked the SaltPepper filter and depending on mode it matches "undot" or "median". There's also the feedback argument but I can check it out later with the feedback plugin from wonkey_monkey.

[kedautinh12]

Real.Finder was added feedback to stab3
https://github.com/realfinder/AVS-Stuff/blob/c6b4a92589842c622c03c7235bcb1d9261e9ba3b/avs%202.5%20and%20up/StabilizationToolsPack.avsi#L423

[Dogway]

Updated Adaptive Sharpen to fix the last bugs and last optimizations. It runs now at 11.40fps on my system with Prefetch(8). I will now try to port CAS, SeeSaw and other sharpeners for a Sharpeners Pack.
If any it made me appreciate more LSFmod and its quality, specially when paired with supersampling.

Also updated SMDegrain with latest changes, that is, the contrasharpening function was fixed to either use spatial or temporal limiting. Also some values for HD were tuned. Should be faster now.
Besides that BM3D prefiltering has been added. Resizing down kernel was changed to bilinear for UHD, and a threads var was fixed when empty.

[kedautinh12]

Thanks, but CAS have avisynth filter, why you still want port it??

[Dogway]

Yes, that's true, forgot about it. If it's too complex I pass otherwise I may add it to the library.

[kedautinh12]

Seesaw have avisynth script
https://github.com/realfinder/AVS-Stuff/blob/Community/avs%202.5%20and%20up/seesaw.avsi

[Dogway]

I'll update my online tool at some point as it now inserts automatic pixel references (when it can) and also has a variable-ised output option.

wonkey_monkey: Is the variable reference output option included in the tool? It's not showing in any of my browsers, it could come very handy for these long sorting networks.

[Julek]

I wonder if it would be possible to do something like OpenCV Adaptive Thresholding with Expr(), it would be useful for mask

[tormento]

contrasharpening function was fixed to either use spatial or temporal limiting.
Which are the best case scenario to choose from spatial or temporal?
Besides that BM3D prefiltering has been added.
Great, I will try it ASAP.

[Dogway]

Julek: I had a look into Otsu's method a month ago but wondered whether it would be useful at all. Well it uses histogram bins so that's not something I can do in Expr but I just implemented a cheap dynamic thresholding to ex_binarize() will upload soon. If Adaptive Thresholding doesn't use histogram bins it can be possible, I will have a look.

tormento: Currently in SMDegrain contrasharpening is stuck to spatial limiting. Spatial limiting is like the defacto method, I like it over temporal for anime to keep faithful lineart. Temporal could be useful for situations like poor interframe correlation like noisy sources, shaky or very action oriented films. One option would be to switch from one to another automatically depending on sequence content (via motion vectors).

[tormento]

tormento
Thanks.

I tried the prefilter=5 and it gives me the error:

Script error: There is no function named 'MatchColorFormat'.
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.2.3d~Dogway.avsi, line 657)
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.2.3d~Dogway.avsi, line 159)

Would you please explain me better what the trymany option does? I have tried to understand thru MVTools manual but it keeps being of obscure meaning to me.

[Dogway]

You need Utils-r41 for MatchColorFormat(). I haven't worked on format conversions lately because Transforms Pack is on halt so for the time being I use internal or other scripts.

trymany I don't know it's a remnant from real.finder's mod. There are still a few things I have to look up.

[zorr]

Trymany is obscure to me as well but my tests indicate (https://forum.doom9.org/showthread.php?p=1945684#post1945684) that trymany=false gives better results. At least on one test case.

[Julek]

Julek: I had a look into Otsu's method a month ago but wondered whether it would be useful at all. Well it uses histogram bins so that's not something I can do in Expr but I just implemented a cheap dynamic thresholding to ex_binarize() will upload soon. If Adaptive Thresholding doesn't use histogram bins it can be possible, I will have a look.

I think Adaptive is more useful than Otsu's, actually I already use Adaptive in Vapoursynth via numpy, but I believe that a direct implementation via Exp (or std.Convolution in VS) would help the speed.

[tormento]

You need Utils-r41 for MatchColorFormat().
I guess for floating point format used by BM3D or what else?

Edit: tried and can't run both in StaxRip and in VirtualDub2. StaxRip plainly hangs, Virtualdub2 spits the following:

https://i1.lensdump.com/i/ZM0iTQ.png

My script:

SetMemoryMax()
SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\1_33 Grosso guaio a Chinatown\grosso.dgi",ct=132,cb=132,cl=0,cr=0)
ConvertBits(16)
SMDegrain (tr=4, thSAD=400, trymany=false, refinemotion=true, contrasharp=false, PreFilter=5, truemotion=true, plane=4, chroma=true)
Prefetch(6)

AVSMeter64 output:

AVSMeter 3.0.8.0 (x64), (c) Groucho2004, 2012-2021
AviSynth+ 3.7.1 (r3431, master, x86_64) (3.7.1.0)

Exception 0xC000001D [STATUS_ILLEGAL_INSTRUCTION]
Module: C:\WINDOWS\System32\KERNELBASE.dll
Address: 0x00007FFB8450467C

No problems whatsoever with prefilter=4.

[DJATOM]

Illegal instruction mostly means that your CPU lacks certain instruction (AVX2 on sandy/ivy bridge?) support.

[tormento]

Illegal instruction mostly means that your CPU lacks certain instruction (AVX2 on sandy/ivy bridge?) support.
Yep, I confirm it's a BM3D test2 issue. With test1 I have no problems at all with a simple test script.

[Dogway]

Sorry I took the day off. Did you include both dll in the plugin folder, both are needed I think?

About MatchColorFormat() it just converts the clip back to input format since BM3D needs YUV444 (for chroma processing). Ideally I would use a personal function from Transforms Pack, but currently it's in WIP state.

[wonkey_monkey]

wonkey_monkey: Is the variable reference output option included in the tool? It's not showing in any of my browsers, it could come very handy for these long sorting networks.

Now updated. It only inserts pixel references for 3x3 or 5x5 networks, with or without the central pixel, so any network with 8, 9, 24, or 25 inputs.

[Dogway]

Thanks a lot, super time saver, I was wasting more than half an hour each time. I needed it to update "medianST" so a 3x3x3, that means 27 inputs, but you can also clip to closest neighbour (26 inputs) and save in this case about 9 lines of min/max.

Something was a bit wrong though (picking output 12 and 13 for 26 inputs 13 layers):
K M dup1 dup1 min K^ max M^
swap1 M P min M^
K N max N^
M N dup1 dup1 min M^ max N^

M N
There's nothing to swap, I replaced with:
K M dup1 dup1 min K^ max M^
M P min M^
K N max N^
M N dup1 dup1 max swap2 min

x swap2 clip

[kedautinh12]

Yep, I confirm it's a BM3D test2 issue. With test1 I have no problems at all with a simple test script.

Cause BM3D_VAggregate() move from BM3DCUDA to BM3DCPU in test2
https://github.com/Dogway/Avisynth-Scripts/blob/1c3ef4f642d71ce9f957192e2a9e9d32e9874a3c/SMDegrain%20v.3.2.3d/SMDegrain%20v3.2.3d.avsi#L653

[wonkey_monkey]

Something was a bit wrong though (picking output 12 and 13 for 26 inputs 13 layers)

I think I've fixed it.

[tormento]

Sorry I took the day off. Did you include both dll in the plugin folder, both are needed I think?
Yep, it was an AVX2 issue.

Please notice that utils-r41 and ZS_shared have duplicate functions in common. I hope it won't give secondary issues.

[kedautinh12]

Just delete function duplicate from utils-r41 or ZS_shared

[tormento]

Sorry I took the day off.
Tested and working with test3 of BM3D.

Now I just have to understand which is better at prefiltering between BM3D and KNLMeansCL, as speeds are comparable when encoding.

Would it be difficult to introduce a function in SMDegrain to show only the differences (as pixels) between the original clip and the resulting one?

[Dogway]

As I could observe BM3D is better at motion estimation and probably also at denoising.

There's a function in ExTools that can be used for comparisons:

ex_makediff(a, aug=true, UV=128)

[kedautinh12]

Yes, BM3D in my opinion is best denoiser

[poisondeathray]

Thanks for the toys Dogway,

Can you check GrainFactory3mod - it appears to do nothing for me, even when using high values for g1str, g2str, g3str and tried different pixel formats. GrainFactory3 ok

colorbars(pixel_type="YV12")
GrainFactory3mod
#GrainFactory3

[Dogway]

poisondeathray: I will have a look. Actually I had plans to replace GrainFactory3mod completely as it shouldn't be that slow. I want to make use of the Addgrain corr args to avoid resizing clip 3 times.

Julek: Do you have an input and output sample? I got some code but I'm not sure what I should target. Also there are adaptive mean/median/gaussian, which one do you need?

[gispos]

Then please also check ex_contrast, parameter "sat" does not work for me.
Thank you.

[Dogway]

poisondeathray: I just checked and it's fine here, using your colorbars call and the GrainFactory3mod call from OP. The call is actually "GrainFactory3mod" not "GrainFactory3" (fixed in OP). Might you be missing a dependency?

gispos: I tested and yes, ex_contrast() defaults to UV=1 for YUV sources. I fixed it by enabling UV=3 if sat is defined, will upload for next update.
UV = Default(UV, Defined(sat) || rgb ? 3 : 1)

In any case the saturation of ex_contrast() isn't meant as a full fledged saturation function, that's the task of ex_vibrance() but I admit it's a bit abandoned lately. Among other things I also need to make it color space aware. ex_contrast() saturation is a simple lerp, more to keep along luma contrast.

[poisondeathray]

poisondeathray: I just checked and it's fine here, using your colorbars call and the GrainFactory3mod call from OP. The call is actually "GrainFactory3mod" not "GrainFactory3" (fixed in OP). Might you be missing a dependency?


Yes, GrainFactory3mod . I mentioned GrainFactory3 because that separate older (non ex) function works

If I copy/paste the function into the script and rename it something like GrainFactory3mod2 , it still returns original clip, so it's not autoloading some other function with the same name

Yes all dependencies are met on the list, with same or newer versions

[gispos]

gispos: I tested and yes, ex_contrast() defaults to UV=1 for YUV sources. I fixed it by enabling UV=3 if sat is defined, will upload for next update.

With UV=3 it works, I hadn't even looked at UV.

[Dogway]

poisondeathray: Maybe you have a function (name) duplication? Another thing to care about avs+ version. Also are you using MIX mod? Both should work though but I always use MIX mods myself mostly. In any case download again from the repo (just updated) and check for dependencies.

ExTools updated to v4.6.
-ex_blur() whole refactor. Now it uses true gaussian coefficients and float radius. For matching removegrain(12), blur(1) use ex_boxblur(mode="weighted")
-ex_smartblur() bugfix that prevented the function from working
-ex_binarize() dynamic mode
-ex_median() optimize "medianST" mode

EX/MIX mods
-Rebased to latest ExTools
-Reverted removegrain(19) to removegrain(20)

I also tried to port removegrainHD smartmedian modes but failed to it, spent whole day yesterday. Might add my attempts if anyone is interested to give it a stab.

[poisondeathray]

poisondeathray: Maybe you have a function (name) duplication? Another thing to care about avs+ version. Also are you using MIX mod? Both should work though but I always use MIX mods myself mostly. In any case download again from the repo (just updated) and check for dependencies.


AVS+ 3.7.1 r3431

Yes I just re-downloaded and check prerequisites. Not sure what's going on

Neither GrainFactory3mod EX or MIX mods work for me. I renamed the functions to call them separately. I even removed autoloading GrainFactory3mod.avsi versions and manually copy/pasted function into script . I double check that function is missing - ie. error message when GrainFactory3Mod is called

eg. rename function

function GrainFactory3mod_ex(clip...

function GrainFactory3mod_mx(clip...

They return the original clip for me

The original GrainFactory3mod from 2020.04.06 (I think real_finder) works ok

[Dogway]

I don't know, there's nothing special on my mod. In these situations I would work my way up within the function to check up where it starts to fail. Output process clip in different stages to see what's failing. Usually it tends to be a dumb oversight. Use simple script, empty the plugin folder, etc.

If you can please try replacing the last line:
use_ALMask ? mt_merge(result... }
With just
result }

[poisondeathray]

I don't know, there's nothing special on my mod. In these situations I would work my way up within the function to check up where it starts to fail. Output process clip in different stages to see what's failing. Usually it tends to be a dumb oversight. Use simple script, empty the plugin folder, etc.

If you can please try replacing the last line:
use_ALMask ? mt_merge(result... }
With just
result }

Same thing; :(

It's ok - Don't worry about it, the original mod still works ok for me

[Julek]

Julek: Do you have an input and output sample? I got some code but I'm not sure what I should target. Also there are adaptive mean/median/gaussian, which one do you need?

Here is an example of mean and gaussian with blockSize=15 and C=3, both applied to luma after a prefilter.
I usually use the mean option more.
Link (https://drive.google.com/drive/folders/18wjz_kEQLvnfmdzoNsXhOFCeR2vLXdn7)

[Dogway]

ExTools updated to v4.7.
-Guidelines for common operations with ex_xxpand(): Opening, closing, smooth, outline, inline, gradient, top hat, black hat.
-ex_hitormiss(). New function for advanced binary mask operations. Modes: Thin, thicken, prune, pruneS, boundary, convexhull, convexhull45, corner, cornerS, skeleton, SKIZ, end and junction. I may add more in the future.

EX/MIX mods
-Minor cosmetic stealth updates

[Dogway]

@Julek: I tried to port adaptive threshold, there are many flavours though but couldn't make anyone work, unless my integral image is wrong I think the problem is in the second part, the thresholding.

It involves two passes, since the integral overflows range_max I'm testing in 32-bits first.
# mean of the local sum
str = "x[-2,-2] A^ x[-1,-2] B^ x[0,-2] C^ x[1,-2] D^ x[2,-2] E^
x[-2,-1] F^ x[-1,-1] G^ x[0,-1] H^ x[1,-1] I^ x[2,-1] J^
x[-2,0] K^ x[-1,0] L^ x[0,0] M^ x[1,0] N^ x[2,0] O^
x[-2,1] P^ x[-1,1] Q^ x[0,1] R^ x[1,1] S^ x[2,1] T^
x[-2,2] U^ x[-1,2] V^ x[0,2] X^ x[1,2] Y^ x[2,2] Z^

A B + WA^ WA C + WB^ WB D + WC^ WC E + WD^
A F + WE^ F G WA + + WF^ F G H WB + + + WG^ I WG WC + + WB - WH^ J WH WD + + WC - WI^
K WE + WJ^ K L WF + + WK^ K L M WG + + + WL^ N WL WH + + WG - WM^ O WM WI + + WH - WN^
P WJ + WO^ P Q WK + + WP^ P Q R WL + + + WQ^ S WQ WM + + WL - WR^ T WR WN + + WM - WS^
U WO + WT^ U V WP + + WU^ U V X WQ + + + WV^ Y WV WR + + WQ - WX^ Z WX WS + + WR - WY^

A WA WB WC WD WE WF WG WH WI WJ WK WL WM WN WO WP WQ WR WS WT WU WV WX WY + + + + + + + + + + + + + + + + + + + + + + + + 25 /"

For the second part I'm not sure whether I should pick the box "vertices", or their range, and if the bottom-right corner should be [x,y] or [x+2,y+2]. In any case I tested all possible combinations (5x5 only, not 15x15 yet) but didn't yield any meaningful results.

Some attempts for thresholding:
str = "x y[2,-2] y[-2,2] + y[-2,-2] - y[2,2] - 25 / 1 0.14 - * <= 0 range_max ? "
str = "x y[2,2] y[-2,-2] + y[-2,2] - y[2,-2] - 25 / 0.8 * <= 0 range_max ? " # this seems legit
str = "x y[0,0] y[-2,-2] + y[-2,0] - y[0,-2] - 9 / 1 0.04 - * <= 0 range_max ? "
str = "x[0,0] y[0,0] y[-2,2] + y[-2,0] - y[0,2] - 9 / 1 0.91 - * <= range_max 0 ? "
str = "y[0,0] y[-2,2] + y[-2,0] - y[0,2] - 9 / 0.003 <= range_max 0 ? " # this seems legit
str = "x y[0,0] y[-1,0] y[-1,-1] y[0,-1] y[-2,-2] + + + + y[-2,0] - y[-2,-1] - y[0,-2] - y[-1,-2] - 9 / 1 0.958 - * <= range_max 0 ? "
str = "x[0,0] y[0,0] y[-1,0] y[-1,-1] y[0,-1] y[-2,-2] y[-2,0] y[-2,-1] y[0,-2] y[-1,-2] y[-2,-2] + + + + + + + + + y[-2,0] - y[-2,-1] - y[0,-2] - y[-1,-2] - 9 / 1 0.9939 - * <= range_max 0 ? "
str = "x y y 25 / A@ - B@ 1 B - / 1 - 0.445 * 1 + A * <= 0 range_max ? " # y should be local sum (without mean)
str = "x x y 25 / A@ - dup * sqrt 128 / 1 - 0.34 * 1 + A * <= 0 range_max ? " # Sauvola (y is local mean -no integral-)
str = "x x y 25 / A@ - dup * sqrt 0.5 / 1 - 0.25 * -10 A * exp 2 * 1 + + A * <= 0 range_max ? " # Phansalkar (y is local mean -no integral-)
I might need to research a bit more.



By the way, for Otsu's I found a way to create histogram bins, well a poor man's version of it, simply threshold image for each bin, subtract the previous thres and average each bin get the amount of values.
ScriptClip("""
thr1=ex_binarize(25.5,true)
bin1=thr1.AverageLuma()
thr2=ex_logic(thr1, ex_binarize(51,true), "andn")
bin2=thr2.AverageLuma()
... """)

[Dogway]

ExTools updated to v4.8.
-ex_binarize() added "otsu" mode.
-ex_median() fixed "adaptive" (for good). Optimized "median5", "medianST", "medianT" and "medianT5"
-ex_repair() optimized "median5"
-ex_bs() upgraded to convert from any bitdepth to any bitdepth, including self bitdepth with different bit scale.
-Updated benchmarks

EX/MIX mods
-Rebase to latest ExTools

SMDegrain updated to v3.2.4d.
-Revert to removegrain(20)
-Fix ex_BM3D() issue when input is 32-bit
-Revert workaround for empty 'threads' issue
-Swapped mt_merge with ex_merge, thus totally removing masktools2 dependency

GradePack updated to v2.6.
-Update ex_bs()
-Optimize ex_blend() "interpolation" mode
-ex_contrast() enable chroma if sat is defined.

[Dogway]

@Tormento: Couldn't refrain myself so I cobbled this up in a moment. Defaults are not tuned, check against STPresso() to see how it compares, this runs about 100fps, STPresso 3.3fps.

# Spatio-Temporal Thresholded Weighted Median

function STTWM(clip a, int "sthres", int "tthres", int "athres", int "slimit", int "tlimit", int "UV", bool "fulls") {

rgb = isRGB(a)
isy = isy(a)
bi = BitsPerComponent(a)

ths = Default(sthres, 24) # 0 - 100%
tht = Default(tthres, 49) # 0 - 100%
tha = Default(athres, 1) # 0 - 100%
sl = Default(slimit, 5) # 0 - 255
tl = Default(tlimit, 5) # 0 - 255
UV = Default(UV, rgb ? 3 : 1)
fs = Default(fulls, rgb)

ts = ths/100.
tt = tht/100.
ta = (100-tha)/100.
rt = 1 - (ts + tt) * ta

Assert(ths+tht<101, "Spatial and Temporal thresholds cannot sum more than 100%.")
Assert(IsVersionOrGreater(3,7,1), "Update AviSynth+ version to 3.7.1+")

str = Format( \
"x[-1,1] x[1,1] dup1 dup1 min K^ max P^
x[-1,-1] x[1,-1] dup1 dup1 min L^ max O^
x[0,1] dup D^ N^
x[-1,0] dup C^ M^
x[0,-1] dup I^ J^
x[1,0] dup G^ H^
x[0,0] dup dup dup A^ F^
B^ E^
N P dup1 dup1 min N^ max P^
F O dup1 dup1 min F^ max O^
J M dup1 dup1 min J^ max M^
I L dup1 dup1 min I^ max L^
B K dup1 dup1 min B^ max K^
E H dup1 dup1 min E^ max H^
D G dup1 dup1 min D^ max G^
A C dup1 dup1 min A^ max C^
H P dup1 dup1 min H^ max P^
M O dup1 dup1 min M^ max O^
E N dup1 dup1 min E^ max N^
C L dup1 dup1 min C^ max L^
G K dup1 dup1 min G^ max K^
F J dup1 dup1 min F^ max J^
A I dup1 dup1 min A^ max I^
B D dup1 dup1 min B^ max D^
O P min O^
L N dup1 dup1 min L^ max N^
H M dup1 dup1 min H^ max M^
J K dup1 dup1 min J^ max K^
D I dup1 dup1 min D^ max I^
F G dup1 dup1 min F^ max G^
C E dup1 dup1 min C^ max E^
A B max B^
M O min M^
K N min K^
H L dup1 dup1 min H^ max L^
G J dup1 dup1 min G^ max J^
E I dup1 dup1 min E^ max I^
C F max F^
B D max D^
K M min K^
E L dup1 dup1 min E^ max L^
H J dup1 dup1 min H^ max J^
G I dup1 dup1 min G^ max I^
D F max F^
K L min K^
I J min I^
G H max H^
E F max F^
I K min I^
F H max H^
H I dup1 dup1 max swap2 min

x swap2 clip S@

dup dup dup
a dup
z dup
y b dup1 dup1 min swap2 max
swap4 swap1 swap8 dup1 dup1 min swap2 max
swap4 swap1 swap6 dup1 dup1 min swap2 max
swap9 swap1 swap3 dup1 dup1 min swap2 max
swap7 swap1 swap5 dup1 dup1 min swap2 max
swap2 swap1 swap4 dup1 dup1 min swap2 max
swap3 swap1 swap6 dup1 dup1 min swap2 max
swap2 swap1 swap7 dup1 dup1 min swap2 max
swap4 swap1 swap8 dup1 dup1 min swap2 max
swap9 swap1 swap3 min
swap8 swap1 swap5 dup1 dup1 min swap2 max
swap2 swap1 swap4 max
swap2 swap1 swap7 dup1 dup1 min swap2 max
swap7 swap1 swap4 dup1 dup1 min swap2 max
swap2 swap1 swap5 dup1 dup1 min swap2 max
swap6 swap1 swap3 dup1 dup1 min swap2 max
swap2 swap1 swap7 min
swap3 swap1 swap4 dup1 dup1 min swap2 max
swap4 swap1 swap5 dup1 dup1 min swap2 max
swap2 swap1 swap6 max
swap3 swap1 swap2 min
swap3 dup1 dup1 min swap2 max
swap2 swap1 swap4 max
swap2 min
swap2 max

x swap2 clip T@
x - abs {tl} > x T ? {tt} *
S x - abs {sl} > x S ? {ts} * + {ta} * x {rt} * + ")

b = a.selectevery(1,-1)
f = a.selectevery(1,+1)
b2 = a.selectevery(1,-2)
f2 = a.selectevery(1,+2)

isy ? Expr(a, b2, b, f, f2, str, optSingleMode = false ) : \
UV == 1 ? Expr(a, b2, b, f, f2, str, "", optSingleMode = false ) : \
Expr(a, b2, b, f, f2, str, ex_UVexpr(str, UV, bi, rgb, fs), optSingleMode = false, scale_inputs="none")

}

[tormento]

Couldn't refrain myself so I cobbled this up in a moment.
I suffer OCD too :D
Defaults are not tuned, check against STPresso() to see how it compares, this runs about 100fps, STPresso 3.3fps.
I just want to replicate

Spresso(10,30).STpresso(10,30,4,22,8,49,1,mc=true)

just a bit faster :o

[Dogway]

I'm trying to port mt_hysteresis() (slow and seems a little buggy on each F5) to ExTools. I'm still not sure this can be ported even by reading the source code (https://github.com/pinterf/masktools/blob/16bit/masktools/filters/mask/hysteresis/hysteresis.cpp).

I managed to get a very similar output with this, which basically is just a 8-connected hit structuring element. The other blocks in red still don't add or help at all.
function ex_hysteresis(clip a, clip b, int "UV", bool "fulls") {

rgb = isRGB(a)
isy = isy(a)
bi = BitsPerComponent(a)

UV = Default(UV, rgb ? 3 : 1)
fs = Default(fulls, rgb)

str = "x[-1,-1] A^ x[0,-1] B^ x[1,-1] C^
x[-1,0] D^ x[0,0] E^ x[1,0] F^
x[-1,1] G^ x[0,1] H^ x[1,1] I^
y[-1,-1] J^ y[0,-1] K^ y[1,-1] L^
y[-1,0] M^ y[0,0] N^ y[1,0] O^
y[-1,1] P^ y[0,1] Q^ y[1,1] R^

J A min
K B min max
L C min max
M D min max
N E min max
O F min max
P G min max
Q H min max
R I min max X^

J
K max
L max
M max
O max
P max
Q max
R max Z^

J N min
K N min max
L N min max
M N min max
O N min max
P N min max
Q N min max
R N min max Y^

Y 0 != range_max 0 ? "

isy ? Expr(a, b, str, optSingleMode=false) : \
UV == 1 ? Expr(a, b, str, "", optSingleMode=false) : \
Expr(a, b, str, ex_UVexpr(str, UV, bi, rgb, fs), optSingleMode=false) }

Reading the VS docs (http://www.vapoursynth.com/doc/plugins/misc.html) says:
"If we treat the planes of the clips as representing 8-neighbourhood undirected 2D grid graphs, for each of the connected components in clipb, the whole component is copied to the output plane if and only if one of its pixels is also marked in the corresponding plane from clipa."

The X^ block tries to compare every corresponding pixel from clipb to clipa, so if it's a hit, then the Y^ 'connected' block can be evaluated and passed to output clip. I also tested by simply comparing E to Y.

[guest]

Hi Dogway,

I just wanted to say that I have been using SMDegrain & ExTools (early builds) successfully in RipBot 264, but with the lasted builds I get an error when running AVSMeter saying there is an issue with Line 446 in SMDegrain.

Are there any other dependancies that need to be used with the later builds ???

I can work around by using older builds of SMDegrain.

Regards

[Dogway]

Maybe if you are doing contrasharpening with values you need my mod of LSFmod, or if you are using BM3D prefiltering you might need Utils-r41.
I noticed that in latest LSFmod I changed 'defaults' arg to 'preset', try changing that in line 337 until I update SMDegrain.

[guest]

Maybe if you are doing contrasharpening with values you need my mod of LSFmod, or if you are using BM3D prefiltering you might need Utils-r41.
I noticed that in latest LSFmod I changed 'defaults' arg to 'preset', try changing that in line 337 until I update SMDegrain.

Thanks for the info, and yes, one of the scripts that throws up the line 446 error IS a contrasharp=30 entry...

Also, I checked the version of LSFmod I was using, and it's was a little old @ v2.187, the latest that I could find was v2.191...doesn't that sound right ??

I just had a look at Line 337 in v2.191, it's blank :eek:

Don't know anything about BM3D....is there a forum or something around ??

I'll do some updates, and let you know :)

OK, back already, and with the very quick response from "videoh" & "kedautinh12", I have some stuff to study :) thanks guys :)

So I now have LSFMod v2.9mx, so I will see what that does.

[videoh]

Don't know anything about BM3D....is there a forum or something around ?? https://forum.doom9.org/showthread.php?t=183066&highlight=BM3D

Read the whole thread. test3 supports AVS+.

[kedautinh12]

New LSFmod
https://github.com/Dogway/Avisynth-Scripts/blob/master/MIX%20mods/LSFmod.v2.9x.avsi
BM3D
http://avisynth.nl/index.php/BM3DCUDA
Edit: oh DG faster in info of BM3D

[guest]

https://forum.doom9.org/showthread.php?t=183066&highlight=BM3D

Read the whole thread. test3 supports AVS+.

:thanks:

[guest]

New LSFmod
https://github.com/Dogway/Avisynth-Scripts/blob/master/MIX%20mods/LSFmod.v2.9x.avsi
BM3D
http://avisynth.nl/index.php/BM3DCUDA
Edit: oh DG faster in info of BM3D

:thanks:

[Dogway]

Just updated SMDegrain to v3.2.5d. Among the mentioned, about updating LSFmod argument I also added an option to disable the previous hard-coded "UHDhalf", for faster UHD processing, by default it's set to true though.
The main updates are moving ex_contrasharpening() to SharpenersPack, optimization of ex_sbr() and big overhaul of ex_MinBlur().

About ex_MinBlur() I did heavy edits for two reasons, faster processing (between 50 and 100% more) and better quality, by replacing removegrain(4) with ex_median("edgeCL") and MedianBlur(2) with ex_median("EMF2"), a custom made median filter that I'm yet to upload with ExTools, probably tomorrow.
Since these replacements protect edges quite well I added some post-blur so lines are not dirty and aiding to denoise for a practical match, therefore respecting the original intention while actually improving on it. I added an argument 'old' to go back to old method.

[StainlessS]

Had a bit of a look at ExTools a bit earlier, outstanding!
Respect Bro.
[wish I understood some of it].

And thanks from here too :)

EDIT: And I learnt a few things just from this

##################
## MORPHOLOGICAL #
##################
##
## Opening: To clean small blotches
## ex_inpand(n).ex_expand(n)
##
## Closing: To close gaps
## ex_expand(n).ex_inpand(n)
##
## Smooth: Opening + Closing, effectively cleaning binary images
## ex_inpand(n).ex_expand(n+n).ex_inpand(n)
##
## Outline: To reveal the outer edge of a shape
## ex_makediff(ex_expand(n), a, dif=false)
##
## Inline: To reveal the inner edge of a shape
## ex_makediff(a, ex_inpand(n), dif=false)
##
## Gradient: To reveal the boundary of elements in a binary image
## ex_makediff(ex_expand(n), ex_inpand(n), dif=false)
## or
## ex_hitormiss(mode="boundary") # thinner
##
## Top Hat: (a - opening) To reveal bright elements over dark backgrounds on a greyscale image
## ex_makediff(a, ex_inpand(n).ex_expand(n), dif=false)
##
## Black Hat: (closing - a) To reveal dark elements over bright backgrounds on a greyscale image
## ex_makediff(ex_expand(n).ex_inpand(n), a, dif=false)
##

All that masking stuff is mostly a mystery to me.

EDIT: I've never known what term to look for as reference for masktools type operations [inpand/expand etc], just found it,
Google "morphological operations"

[guest]

Maybe if you are doing contrasharpening with values you need my mod of LSFmod, or if you are using BM3D prefiltering you might need Utils-r41.
I noticed that in latest LSFmod I changed 'defaults' arg to 'preset', try changing that in line 337 until I update SMDegrain.

Well, I've been "fiddling" most of the day, and I'm just confused...

There are too many SMDegrain builds out there, yesterday it was 3.2.4d, today it's 3.2.5d, there's different variants of LSFMod...the ex & mx "series".

It would be great to know ALL the dependencies required for these.

And I have found that depending on the prefilter option I choose, to what SMDegrain error gets thrown up.

I was trying to get BM3D to work, and that seems to be prefilter #5, but that shows an error with Line 158 (I think) inputP, then you change the prefilter to another number, and then there's an error with Line 448 which is just 2 of these {.

There's Utils r41, that doesn't help...

And I couldn't find this :- I noticed that in latest LSFmod I changed 'defaults' arg to 'preset', try changing that in line 337 until I update SMDegrain.

Anyway, most of this it WAY over my head, so maybe I just need to stick with what works, and leave it at that. :(

[Dogway]

@Pauly Dunne: It works for me so I think the scripts are fine. You have got suggestions from others to update to latest avisynth+ and scripts. About ex and mix mods, it's described in OP. If you want raw speed go to mix mods, ex mods usually just remove the dependency of RGTools and masktools2 at the cost of some speed.

For my scripts the dependencies almost go to the scripts in OP; ExTools, Transforms Pack and Resizers Pack, and externally Zs_Shared and Utils-r41 (will get rid of this dependency soon with TransformsPack). Then you have all the plugins which really are not many, just the typical; RGTools, masktools2, mvtools2, GRunT, and then depending on each script (ie. DFTTest, KNLMeansCL, BM3D, AddgrainC...). That's the purpose of this project, remove plugin dependencies to the max.

I can only vouch for my scripts like SMDegrain or mods like EX/MIX mods. I came back here 3 months ago after being away for a few years so it's normal you see some changes. When I finish everything in a few months you won't see changes anymore. Bug reports aside, ExTools is one version off of being finished, SMDegrain is finished today. I won't work on these anytime soon and will focus on TransformsPack only.

I just updated ExTools and SMDegrain. Try it and if you find an error report back with the error and filter call.

[Dogway]

@StainlessS: Thanks! The sentiment is mutual lol. Actually I learn as I go. The most intriguing for me was the hit-or-miss functions, really I spent days trying to figure out how to translate that to mach... Expr code. Was kind of a feat for me. Anyway I couldn't port Hysteresis -__- but ex_luts is shaping up nicely, it's in a working state for some modes. With some tests it shows between double or three times faster than mt_luts(). Pixel Addressing was a hidden gem, hope it gets some optimization love further down the line.

I updated the morphological guide with one extra addition, pseudomedian:

##################
## MORPHOLOGICAL #
##################
##
## Opening: To clean small blotches
## ex_inpand(n).ex_expand(n)
##
## Closing: To close gaps
## ex_expand(n).ex_inpand(n)
##
## Smooth: Opening & Closing, effectively cleaning binary images
## ex_inpand(n).ex_expand(n+n).ex_inpand(n)
##
## Outline: To reveal the outer edge of a shape
## ex_makediff(ex_expand(n), a, dif=false)
## or
## ex_hitormiss(mode="outline") # faster
##
## Inline: To reveal the inner edge of a shape
## ex_makediff(a, ex_inpand(n), dif=false)
## or
## ex_hitormiss(mode="inline") # faster
##
## Gradient Magnitude: To reveal the boundary of elements in a binary image
## ex_makediff(ex_expand(n), ex_inpand(n), dif=false)
## or
## ex_edge("min/max") # faster
## or
## ex_hitormiss(mode="boundary")
##
## Top Hat: (a - opening) To reveal bright elements over dark backgrounds on a greyscale image
## ex_makediff(a, ex_inpand(n).ex_expand(n), dif=false)
##
## Black Hat: (closing - a) To reveal dark elements over bright backgrounds on a greyscale image
## ex_makediff(ex_expand(n).ex_inpand(n), a, dif=false)
##
## Pseudomedian: (opening + closing)/2 Approximation to a simple median filter with morphological structuring elements
## Merge(ex_inpand(n).ex_expand(n), ex_expand(n).ex_inpand(n))
##

[kedautinh12]

@Dogway you have plane to port mt_xxpand_multi.avsi to ex_ family???

[Dogway]

Just do ex_xxpand(3), it's faster.

[kedautinh12]

But ex_xxpand don't have mode="ellipse"

[Dogway]

Well, ex_xxpand don't have mode="ring" ;)

'ellipse' is being worked out in ex_shapes() (forgot about it before declaring the function 'final') so you can do something like:

ex_luts(mode="max",pixels=ex_shapes(2,1,mode="disk"))

I won't add anisotropic/anamorphic shapes to ex_xxpand()

[kedautinh12]

If i'll change mt_lutxy, it can be same with ex_lutxy??
https://github.com/realfinder/AVS-Stuff/blob/e38249397b6cefae2f78932654e0b69ab892491e/avs%202.5%20and%20up/F3KDB_s.avsi#L236

[Dogway]

It isn't using an anamorphic radius, you can use ex_xxpand(), here the function:
function F3KDB_3_masking(clip c, float "mask_thr", int "mask_radius", bool "fulls") {

rgb = isRGB(c)
bi = BitsPerComponent(c)

mthr = Default(mask_thr, 2)
mrad = Default(mask_radius, 2)
fs = Default(fulls, rgb)

Assert(mrad > 0, "F3KDB_3_masking: mask_radius must be more than 0")

thr_lo = ex_bs(max(mthr * 0.75, 1) - 0.0001, 8, bi, fulls=fs)
thr_hi = ex_bs(max(mthr, 1) + 0.0001, 8, bi, fulls=fs)

c
mrad > 1 ? ex_lutxy (ex_expand(mrad,mode="circle"), ex_inpand(mrad,mode="circle"), "x y -",clamp_float=true,fulls=fs) : mt_edge (mode="min/max", thY1=0, thY2=255)
ex_lut (Format("x {thr_lo} - {thr_hi} {thr_lo} - / 0 1 clip range_max *"), fulls=fs)
removegrain (22, -1)
mrad > 1 ? removegrain (12, -1) : last
mrad > 2 ? removegrain (20, -1) : last }


EDIT: By the way, did you notice? That line is Gradient Magnitude from my morphological guide above, you can optimize that further by using ex_luts(), wait a min.

EDIT: Optimized further (58% gain) (still some issues for ex_luts() when using high radii)
function F3KDB_3_masking(clip c, float "mask_thr", int "mask_radius", bool "fulls") {

rgb = isRGB(c)
bi = BitsPerComponent(c)

mthr = Default(mask_thr, 2)
mrad = Default(mask_radius, 2)
fs = Default(fulls, rgb)

Assert(mrad > 0, "F3KDB_3_masking: mask_radius must be more than 0")

thr_lo = ex_bs(max(mthr * 0.75, 1) - 0.0001, 8, bi, fulls=fs)
thr_hi = ex_bs(max(mthr, 1) + 0.0001, 8, bi, fulls=fs)

c
mrad > 1 ? ex_luts(mode="range",pixels=ex_shape(mrad,shape="circle"),clamp_float=true,fulls=fs) : \
mt_edge (mode="min/max", thY1=0, thY2=255)
ex_lut (Format("x {thr_lo} - {thr_hi} {thr_lo} - / 0 1 clip range_max *"), fulls=fs)
removegrain (22, -1)
mrad > 1 ? removegrain (12, -1) : last
mrad > 2 ? removegrain (20, -1) : last }

[guest]

@Pauly Dunne: It works for me so I think the scripts are fine. You have got suggestions from others to update to latest avisynth+ and scripts. About ex and mix mods, it's described in OP. If you want raw speed go to mix mods, ex mods usually just remove the dependency of RGTools and masktools2 at the cost of some speed.

For my scripts the dependencies almost go to the scripts in OP; ExTools, Transforms Pack and Resizers Pack, and externally Zs_Shared and Utils-r41 (will get rid of this dependency soon with TransformsPack). Then you have all the plugins which really are not many, just the typical; RGTools, masktools2, mvtools2, GRunT, and then depending on each script (ie. DFTTest, KNLMeansCL, BM3D, AddgrainC...). That's the purpose of this project, remove plugin dependencies to the max.

I can only vouch for my scripts like SMDegrain or mods like EX/MIX mods. I came back here 3 months ago after being away for a few years so it's normal you see some changes. When I finish everything in a few months you won't see changes anymore. Bug reports aside, ExTools is one version off of being finished, SMDegrain is finished today. I won't work on these anytime soon and will focus on TransformsPack only.

I just updated ExTools and SMDegrain. Try it and if you find an error report back with the error and filter call.

Hi Dogway, thanks for the great explanation, but like I said this is all way over my head...

Not sure what you use all these with to test & use, and I am trying to incorporate more up to date filters to RipBot264, which is a pretty closed app, for the most part.

I have had a fair bit of success, and the first builds of Extools, and earlier builds of SMDegrain worked very well, I think SMDegrain does a lot better job than MDegrain...but the last few builds have just messed the whole thing up :(

Maybe I just need to back track, and use what use to work, and stick with that.

I will try the newer ExTools & SMDegrain, and comment back :)

[kedautinh12]

Try avisynth info tool to check your error
https://forum.doom9.org/showthread.php?t=176079

[guest]

Try avisynth info tool to check your error
https://forum.doom9.org/showthread.php?t=176079

Thanks, I will give that a shot....I've been using AVSMeter to show what line the error is, but that doesn't always makes sense.

Yeah, I can't make any sense of the info this tool provides, it only seems to check what is on the C:/ drive, and installed within the Avisynth folder...

[guest]

Hi Dogway, thanks for the great explanation, but like I said this is all way over my head...

Not sure what you use all these with to test & use, and I am trying to incorporate more up to date filters to RipBot264, which is a pretty closed app, for the most part.

I have had a fair bit of success, and the first builds of Extools, and earlier builds of SMDegrain worked very well, I think SMDegrain does a lot better job than MDegrain...but the last few builds have just messed the whole thing up :(

Maybe I just need to back track, and use what use to work, and stick with that.

I will try the newer ExTools & SMDegrain, and comment back :)

OK, after some interesting testing, this is what I've got:-

This is how the scripts are, in RipBot264 !!!

With this script:-

#Custom
LoadPlugin("%AVISYNTHPLUGINS%\mvtools\mvtools2.dll")
LoadPlugin("%AVISYNTHPLUGINS%\RgTools\RgTools.dll")
Import("%AVISYNTHPLUGINS%\scripts\extras\ExTools52.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\SMDegrain324d.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Zs_RF_Shared.avs")
video=SMDegrain(video,tr=6,thSAD=600,contrasharp=true,refinemotion=true)

Seems to work well, and if I change to SMDegrain 3.2.5d, I get an error @ line 449 in AVSMeter.

Adding LSFmod doesn't make any difference. (meaning no errors)

And this script:-

#Custom
LoadPlugin("%AVISYNTHPLUGINS%\mvtools\mvtools2.dll")
LoadPlugin("%AVISYNTHPLUGINS%\RgTools\RgTools.dll")
Import("%AVISYNTHPLUGINS%\scripts\extras\LSFmod29ex.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\ExTools52.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\SMDegrain325d.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Zs_RF_Shared.avs")
video=SMDegrain(video,tr=6,thSAD=600,contrasharp=30,prefilter=2,str=1.2,refinemotion=true)

If I change this back to SMDegrain 3.2.4d, I get an error @ line 446. in AVSMeter.

If I remove LSFmod, I then get an error @ line 449, in 3.2.5d.

And just for good measure:-

#Custom
LoadPlugin("%AVISYNTHPLUGINS%\mvtools\mvtools2.dll")
LoadPlugin("%AVISYNTHPLUGINS%\RgTools\RgTools.dll")
Import("%AVISYNTHPLUGINS%\scripts\extras\LSFmod29ex.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\ExTools52.avs")
LoadPlugin("%AVISYNTHPLUGINS%\KNLMeansCL\KNLMeansCL.dll")
Import("%AVISYNTHPLUGINS%\scripts\extras\SMDegrain325d.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Zs_RF_Shared.avs")
video=SMDegrain(video,tr=6,thSAD=600,contrasharp=30,prefilter=4,str=1.2,refinemotion=true)


Works OK too, until..(same as above)

So it would be good to know why the first script doesn't like 3.2.5d, and I haven't got a clue how to fix that :(

[kedautinh12]

Maybe you don't have enough memory, how much gb in your rams??
https://github.com/Dogway/Avisynth-Scripts/blob/5d60a5dedaea2f77b75322a6bf2dc2759607a311/SMDegrain%20v.3.2.5d/SMDegrain%20v3.2.5d.avsi#L446

[guest]

Maybe you don't have enough memory, how much gb in your rams??
https://github.com/Dogway/Avisynth-Scripts/blob/5d60a5dedaea2f77b75322a6bf2dc2759607a311/SMDegrain%20v.3.2.5d/SMDegrain%20v3.2.5d.avsi#L446

Wow, I don't understand why that could be an issue.....

I've only got 32Gb, 3600Mhz DDR4.

[kedautinh12]

Wow, I don't understand why that could be an issue.....

I've only got 32Gb, 3600Mhz DDR4.

Can you post your log with AVSMeter or avisynth info tool??

[Dogway]

This is my plugins list, and I have no problems with any of your examples.

ConvertStacked.dll
DirectShowSource.dll
ExTools.avsi
FFMS2.avsi
ffms2.dll
ffms2.lib
ffmsindex.exe
ImageSeq.dll
KNLMeansCL.dll
LSFmod.v2.9x.avsi
masktools2.dll
MtModes.r851.avsi
mvtools2.dll
RgTools.dll
SharpenersPack.avsi
Shibatch.dll
SMDegrain v3.2.5d.avsi
TimeStretch.dll
Utils-r41.avsi
Zs_RF_Shared.avsi

I think your problem is something is not updated, namely Zs_RF_Shared? Update to V1.154.

[guest]

Can you post your log with AVSMeter or avisynth info tool??

Not too sure how to do that...I'll have a look tomorrow.

:thanks:

[guest]

This is my plugins list, and I have no problems with any of your examples.

ConvertStacked.dll
DirectShowSource.dll
ExTools.avsi
FFMS2.avsi
ffms2.dll
ffms2.lib
ffmsindex.exe
ImageSeq.dll
KNLMeansCL.dll
LSFmod.v2.9x.avsi
masktools2.dll
MtModes.r851.avsi
mvtools2.dll
RgTools.dll
SharpenersPack.avsi
Shibatch.dll
SMDegrain v3.2.5d.avsi
TimeStretch.dll
Utils-r41.avsi
Zs_RF_Shared.avsi

I think your problem is something is not updated, namely Zs_RF_Shared? Update to V1.154.

Excellent...yes I think I am running v1.153.

Will keep you posted, and thankyou so much for your help :)

[kedautinh12]

-avsinfo in AVSMeter or you can read here for more details
https://forum.doom9.org/showthread.php?t=174797

[guest]

-avsinfo in AVSMeter or you can read here for more details
https://forum.doom9.org/showthread.php?t=174797

I generally run it within RipBot, so I'm not sure if there are any logs generated, or where they go.

I guess I could run it stand alone (could I ?)

[StainlessS]

on command line just type path and name eg d:\avsmeter.exe
and produces eg
https://i.postimg.cc/Pfpdc8gn/Untitled-00.png (https://postimg.cc/Jt8wHt7p)

EDIT: I have "C:\BIN" in my environment variables PATH, so I don't have to put the path the execuatable of avsmeter.
I store my cmd exe's in C:\BIN

https://i.postimg.cc/FRbVsKqr/Untitled-00.png (https://postimages.org/)

[guest]

Hi StainlessS,

But the only info I can get this to display is in the default installation folder, NOT where I have all the "custom" info for RipBot :(

Got it work, had to copy my custom stuff into the default folder....didn't show any problems, that I could see.

[guest]

This is my plugins list, and I have no problems with any of your examples.

ConvertStacked.dll
DirectShowSource.dll
ExTools.avsi
FFMS2.avsi
ffms2.dll
ffms2.lib
ffmsindex.exe
ImageSeq.dll
KNLMeansCL.dll
LSFmod.v2.9x.avsi
masktools2.dll
MtModes.r851.avsi
mvtools2.dll
RgTools.dll
SharpenersPack.avsi
Shibatch.dll
SMDegrain v3.2.5d.avsi
TimeStretch.dll
Utils-r41.avsi
Zs_RF_Shared.avsi

I think your problem is something is not updated, namely Zs_RF_Shared? Update to V1.154.

Hey Dogway,

Well, unfortunately, I was already using v1.154 of Zs RF, so that hasn't fixed the problem.

Now I shouldn't need to run all those plugins you've listed above, but maybe I should add them, and see what happens.


edit :- OK, I've had a win :) x 1,000....I went thru all your list, and the only one that I hadn't tried was "SharpenersPack.avsi", so I popped that in, and away it went with SMDegrain 3.2.5d.

So now I need to is clean up the mess I've made of all my files & folders, and I should be good to go, and "kedautinh12" gave me a good idea to use avsi's, instead of avs's.

And I'd like to now figure out how to incorporate BM3D....but isn't that only for nVidia GPU's ??

[guest]

-avsinfo in AVSMeter or you can read here for more details
https://forum.doom9.org/showthread.php?t=174797

This is the best I could do....

https://i.imgur.com/BcJ9Kt6.jpg

[kedautinh12]

Hi StainlessS,

But the only info I can get this to dispaly is in the default installation folder, NOT where I have all the "custom" info for RipBot :(

Transport your filters to default installation folder and check with AVSMeter again

[guest]

Transport your filters to default installation folder and check with AVSMeter again

Wow, OK, I'll try that, but there's so many of them...

But what is this going to prove ??

It's strange that the scripts I sent Dogway worked for him...

[kedautinh12]

To checked what errors in your scripts/filter. And you don't need call them anymore in ripbot (don't need LoadPlugin/Import anymore)

[guest]

To checked what errors in your scripts/filter. And you don't need call them anymore in ripbot (don't need LoadPlugin/Import anymore)

So you're saying I don't need these anymore ???

#Custom
LoadPlugin("%AVISYNTHPLUGINS%\mvtools\mvtools2.dll")
LoadPlugin("%AVISYNTHPLUGINS%\RgTools\RgTools.dll")
Import("%AVISYNTHPLUGINS%\scripts\extras\LSFmod29ex.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\ExTools52.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\SMDegrain325d.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Zs_RF_Shared.avs")
video=SMDegrain(video,tr=6,thSAD=600,contrasharp=30,prefilter=2,str=1.2,refinemotion=true)

I can't see that working, that's the way RipBot is built

[guest]

To checked what errors in your scripts/filter. And you don't need call them anymore in ripbot (don't need LoadPlugin/Import anymore)

So far all it's displaying is the filters & scripts info....no other, no errors

[kedautinh12]

So far all it's displaying is the filters & scripts info....no other, no errors

Don't have functions duplicate??

[kedautinh12]

So you're saying I don't need these anymore ???

#Custom
LoadPlugin("%AVISYNTHPLUGINS%\mvtools\mvtools2.dll")
LoadPlugin("%AVISYNTHPLUGINS%\RgTools\RgTools.dll")
Import("%AVISYNTHPLUGINS%\scripts\extras\LSFmod29ex.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\ExTools52.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\SMDegrain325d.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Zs_RF_Shared.avs")
video=SMDegrain(video,tr=6,thSAD=600,contrasharp=30,prefilter=2,str=1.2,refinemotion=true)

I can't see that working, that's the way RipBot is built

Yeah, i use megui with autoload install folder and don't need use call them with LoadPlugin/Import anymore cause they autoload by avisynth in autoload folder when i put them in it.
Btw, change .avs to .avsi, avs+ only autoload .avsi

[guest]

Don't have functions duplicate??

None that I noticed...

[guest]

Yeah, i use megui with autoload install folder and don't need use call them with LoadPlugin/Import anymore cause they autoload by avisynth in autoload folder when i put them in it.
Btw, change .avs to .avsi, avs+ only autoload .avsi

So if "everything" was in one folder, with the .avs's changed to .avsi, and I got the path correct, that would work ???

Could you give me an example of how you've addressed it ??

But what is the advantage of that ??

[kedautinh12]

Example: your script

LoadPlugin("%AVISYNTHPLUGINS%\mvtools\mvtools2.dll")
LoadPlugin("%AVISYNTHPLUGINS%\RgTools\RgTools.dll")
Import("%AVISYNTHPLUGINS%\scripts\extras\LSFmod29ex.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\ExTools52.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\SMDegrain325d.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Zs_RF_Shared.avs")
video=ffms("...")
video=SMDegrain(video,tr=6,thSAD=600,contrasharp=30,prefilter=2,str=1.2,refinemotion=true)

After you put all plugins/scripts in one autoload folder, you only use this script and it will work

ffms("...")
SMDegrain(tr=6,thSAD=600,contrasharp=30,prefilter=2,str=1.2,refinemotion=true)


That is what i do with megui

[guest]

Example: your script

LoadPlugin("%AVISYNTHPLUGINS%\mvtools\mvtools2.dll")
LoadPlugin("%AVISYNTHPLUGINS%\RgTools\RgTools.dll")
Import("%AVISYNTHPLUGINS%\scripts\extras\LSFmod29ex.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\ExTools52.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\SMDegrain325d.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Zs_RF_Shared.avs")
video=ffms("...")
video=SMDegrain(video,tr=6,thSAD=600,contrasharp=30,prefilter=2,str=1.2,refinemotion=true)

After you put all plugins/scripts in one autoload folder, you only use this script and it will work

ffms("...")
SMDegrain(tr=6,thSAD=600,contrasharp=30,prefilter=2,str=1.2,refinemotion=true)


That is what i do with megui

Hmmm,

Not sure that will work for RipBot...

So, I sort of understand the avs to avsi thing, but how to the dll's "auto load" ??

And what if I don't use FFMS ??

RipBot uses L-Smash as it's default decoder, if I'm understanding the ffms reference.

[guest]

Hello again, Dogway,

With the success I've had (see post #236, if you haven't already), I need a little bit more help with the following :-

And does it matter what order they are called ??

#Custom
LoadPlugin("%AVISYNTHPLUGINS%\mvtools\mvtools2.dll")
LoadPlugin("%AVISYNTHPLUGINS%\BM3D\x64\BM3DCUDA_AVS.dll")
LoadPlugin("%AVISYNTHPLUGINS%\BM3D\x64\BM3D_VAggregate_AVS.dll")
LoadPlugin("%AVISYNTHPLUGINS%\RgTools\RgTools.dll")
Import("%AVISYNTHPLUGINS%\scripts\extras\ExTools52.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Sharpeners Pack v0.5.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\SMDegrain325d.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Utils-r41.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Zs_RF_Shared.avs")
video=SMDegrain(video,tr=4,thSAD=400,contrasharp=true,refinemotion=true,prefilter=5)

This BM3D script works (it's slow), but if I change it to BM3DCPU_AVS.dll, it has a error at Line 159 of SMDegrain. (inputP)

I have an AVX2 CPU...5950X

And now the only pre-filter I'm having a problem with is #3 (prefilter=3)

0, 1, 2, 4 (with KNL in there), & 5 (@ CUDA).

What is missing for #3 ??

pref = !GlobalR ? preclip ? prefilter : \
(prefilter==-1) ? inputP : \
(prefilter== 0) ? inputP.ex_MinBlur(0,Chr,fulls=fs) : \
(prefilter== 1) ? inputP.ex_MinBlur(1,Chr,fulls=fs) : \
(prefilter== 2) ? inputP.ex_MinBlur(2,Chr,fulls=fs) : \
(prefilter== 3) ? ex_merge(dfttest(inputP,sstring="0.0:4.0 0.2:9.0 1.0:15.0",tbsize=1,U=chroma,V=chroma,dither=1,threads=1),inputP, \
inputP.ex_lut(Format("x ymin < range_max x {Lthres} > 0 range_max x ymin - range_max {Lthres} ymin - / * - ? ?"),UV=1,fulls=fs), luma=chroma, UV=chr, fulls=fs) : \
(prefilter== 4) ? inputP.ex_KNLMeansCL(device_type="GPU", device_id=device_id, chroma=chroma, a=1,d=1,h=7.0) : \
(prefilter== 5) ? inputP.ex_BM3D(sigma=5,radius=1,UV=Chr) : \
Assert(false, "prefilter must be between -1~5: "+string(prefilter)) : \
inputP

[kedautinh12]

SMDegrain only use BM3D_CUDA
https://github.com/Dogway/Avisynth-Scripts/blob/5d60a5dedaea2f77b75322a6bf2dc2759607a311/SMDegrain%20v.3.2.5d/SMDegrain%20v3.2.5d.avsi#L619

If you want use BM3D_CPU just change line above from BM3D_CUDA(sigma=[s,ch?s/2:0,ch?s/2:0], radius=r, chroma=ch, fast=true, extractor_exp=6) to BM3D_CPU(sigma=[s,ch?s/2:0,ch?s/2:0], radius=r, chroma=ch)

If you want more speed of BM3D try use prefetch(). More details:
https://forum.doom9.org/showthread.php?p=1948420#post1948420
http://avisynth.nl/index.php/SetFilterMTMode

Example:

#Custom
LoadPlugin("%AVISYNTHPLUGINS%\mvtools\mvtools2.dll")
LoadPlugin("%AVISYNTHPLUGINS%\BM3D\x64\BM3DCUDA_AVS.dll")
LoadPlugin("%AVISYNTHPLUGINS%\BM3D\x64\BM3D_VAggregate_AVS.dll")
LoadPlugin("%AVISYNTHPLUGINS%\RgTools\RgTools.dll")
Import("%AVISYNTHPLUGINS%\scripts\extras\ExTools52.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Sharpeners Pack v0.5.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\SMDegrain325d.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Utils-r41.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Zs_RF_Shared.avs")
video=SMDegrain(video,tr=4,thSAD=400,contrasharp=true,refinemotion=true,prefilter=5)
Prefetch(16) #or try any number can make a good speed fps


Edit: for prefilter=3 you forgot LoadPlugin(".../DFTTest.dll")

[guest]

SMDegrain only use BM3D_CUDA
https://github.com/Dogway/Avisynth-Scripts/blob/5d60a5dedaea2f77b75322a6bf2dc2759607a311/SMDegrain%20v.3.2.5d/SMDegrain%20v3.2.5d.avsi#L619

If you want use BM3D_CPU just change line above from BM3D_CUDA(sigma=[s,ch?s/2:0,ch?s/2:0], radius=r, chroma=ch, fast=true, extractor_exp=6) to BM3D_CPU(sigma=[s,ch?s/2:0,ch?s/2:0], radius=r, chroma=ch)

If you want more speed of BM3D try use prefetch(). More details:
https://forum.doom9.org/showthread.php?p=1948420#post1948420
http://avisynth.nl/index.php/SetFilterMTMode

Example:

#Custom
LoadPlugin("%AVISYNTHPLUGINS%\mvtools\mvtools2.dll")
LoadPlugin("%AVISYNTHPLUGINS%\BM3D\x64\BM3DCUDA_AVS.dll")
LoadPlugin("%AVISYNTHPLUGINS%\BM3D\x64\BM3D_VAggregate_AVS.dll")
LoadPlugin("%AVISYNTHPLUGINS%\RgTools\RgTools.dll")
Import("%AVISYNTHPLUGINS%\scripts\extras\ExTools52.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Sharpeners Pack v0.5.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\SMDegrain325d.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Utils-r41.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Zs_RF_Shared.avs")
video=SMDegrain(video,tr=4,thSAD=400,contrasharp=true,refinemotion=true,prefilter=5)
Prefetch(16) #or try any number can make a good speed fps


Edit: for prefilter=3 you forgot LoadPlugin(".../DFTTest.dll")

You're a very informative person :)

Yes, I have that version of Dogway's SMDegrain.

I will add DFTTest, thanks.

dfttest didn't work for pre filter 3, still line 159 error :(, but it needs these installed http://avisynth.nl/index.php/Dfttest..... http://www.fftw.org/install/windows.html

And I think the MT function is handled differently in RipBot scripts.

Have you ever tried RipBot ???

I had a look at MeGUI, seems very basic, and not that up to date (inside)

[kedautinh12]

Are you try that scripts and check the speed??

#Custom
LoadPlugin("%AVISYNTHPLUGINS%\mvtools\mvtools2.dll")
LoadPlugin("%AVISYNTHPLUGINS%\BM3D\x64\BM3DCUDA_AVS.dll")
LoadPlugin("%AVISYNTHPLUGINS%\BM3D\x64\BM3D_VAggregate_AVS.dll")
LoadPlugin("%AVISYNTHPLUGINS%\RgTools\RgTools.dll")
Import("%AVISYNTHPLUGINS%\scripts\extras\ExTools52.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Sharpeners Pack v0.5.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\SMDegrain325d.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Utils-r41.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Zs_RF_Shared.avs")
video=SMDegrain(video,tr=4,thSAD=400,contrasharp=true,refinemotion=true,prefilter=5)
Prefetch(16) #or try any number can make a good speed fps

[Dogway]

@Pauly Dunne: Can you try this version (https://github.com/Dogway/Avisynth-Scripts/blob/master/SMDegrain%20v.3.2.6d/SMDegrain%20v3.2.6d.avsi). I fixed some UHDhalf and interlacing logic. Maybe your clip was different than what I tested on.

[kedautinh12]

Download here for new ver fftw 3.3.8
https://forum.doom9.org/showthread.php?t=174470

Benchmark between of fftw3 compilers
https://forum.doom9.org/showthread.php?p=1911115#post1911115

For ICL & GCC you need Intel C++ too
https://software.intel.com/content/www/us/en/develop/articles/intel-compilers-redistributable-libraries-by-version.html

Edit: after downloaded, you can put fftw3.dll or libfftw3f-3.dll in Windows/system32 (for x64 ver) and Windows/sysWOW64 (for x86 ver)

[guest]

Download here for new ver fftw 3.3.8
https://forum.doom9.org/showthread.php?t=174470

Benchmark between of fftw3 compilers
https://forum.doom9.org/showthread.php?p=1911115#post1911115

For ICL & GCC you need Intel C++ too
https://software.intel.com/content/www/us/en/develop/articles/intel-compilers-redistributable-libraries-by-version.html

Edit: after downloaded, you can put fftw3.dll or libfftw3f-3.dll in Windows/system32 (for x64 ver) and Windows/sysWOW64 (for x86 ver)

Yes, I do have them, and I got it to work, so I will check version numbers, thanks

[guest]

@Pauly Dunne: Can you try this version (https://github.com/Dogway/Avisynth-Scripts/blob/master/SMDegrain%20v.3.2.6d/SMDegrain%20v3.2.6d.avsi). I fixed some UHDhalf and interlacing logic. Maybe your clip was different than what I tested on.

I'm now getting a Line 160 error, when I change BM3D, CUDA to CPU, when using prefilter=5.

But 3.2.6d works on the other scripts / filters. :)

[kedautinh12]

Megui tools up to date if you change sever to development
https://forum.doom9.org/showthread.php?p=1946568#post1946568

[guest]

Megui tools up to date if you change sever to development
https://forum.doom9.org/showthread.php?p=1946568#post1946568

Fair enough, but I don't think I need to learn how to use another app.

But have you had any experience with RipBot ??

[kedautinh12]

I'm now getting a Line 160 error, when I change BM3D, CUDA to CPU, when using prefilter=5.

But 3.2.6d works on the other scripts / filters. :)

You changed command in this line or just change LoadPlugin??
https://github.com/Dogway/Avisynth-Scripts/blob/19f0b90838e0a7022fbf9c698c71ff201f7765bf/SMDegrain%20v.3.2.6d/SMDegrain%20v3.2.6d.avsi#L617

[kedautinh12]

Fair enough, but I don't think I need to learn how to use another app.

But have you had any experience with RipBot ??

No, i'm just experience with use scripts in avs+ and i think 2 app (ripbot & megui) work same scripts structure with avs+

[guest]

You changed command in this line or just change LoadPlugin??
https://github.com/Dogway/Avisynth-Scripts/blob/19f0b90838e0a7022fbf9c698c71ff201f7765bf/SMDegrain%20v.3.2.6d/SMDegrain%20v3.2.6d.avsi#L617

I just changed the whole thing !!!

[guest]

No, i'm just experience with use scripts in avs+ and i think 2 app (ripbot & megui) work same scripts structure with avs+

Well, I only use RipBot, and I have only just recently started to figure out how scripts work...a LOT of trial & error, and help from guy's like you, & Dogway, etc.

The main reason I started was RipBot wasn't getting any regular updates, like it use to get, not sure why, maybe the dev "Atak" has lost interest.

It's rather out of date, now :(...still works well, especially the Distributed Encoding function.

Cheers

[guest]

If you want use BM3D_CPU just change line above from BM3D_CUDA(sigma=[s,ch?s/2:0,ch?s/2:0], radius=r, chroma=ch, fast=true, extractor_exp=6) to BM3D_CPU(sigma=[s,ch?s/2:0,ch?s/2:0], radius=r, chroma=ch)


Hi, I read somewhere that if you want to use the BM3D_CPU option, you don't need BM3D_VAggregate, is that correct ??? (haven't had a chance to test it for myself)

[kedautinh12]

Hi, I read somewhere that if you want to use the BM3D_CPU option, you don't need BM3D_VAggregate, is that correct ??? (haven't had a chance to test it for myself)

No, bm3d.VAggregate should be called after temporal filtering
https://github.com/WolframRhodium/VapourSynth-BM3DCUDA#notes

[Dogway]

Very good news, advancements in ex_luts(), finally managed to port "*.mp4 guy" DeStripe() filter, I post the benchmarks.
Normally with plain pixel fetching Prefetch(4) (cores in my case) is faster, when expressions are involved I get more speed with Prefetch(6)

# P(4) 29 P(6) 36.5 (8-bit)
# P(4) 244 P(6) 271 (8-bit)
# P(4) 179 P(6) 184 (16-bit)

Original DeStripe() in 16-bits doesn't even start at all. With this I finally can port more filters that make use of ex_luts() like some other sharpeners. Will upload ExTools on Monday most likely. Just sharing the excitement : P


By the way, is there any way to retrieve Cores/Threads so I can automatically set correct Prefetch() for some functions/modes?

[kedautinh12]

Your destripe() ported need cullresize.dll same destripe in vs (.py), avs (.avsi) or don't need anymore??
https://github.com/YomikoR/VapourSynth-Destripe#related

[Reel.Deel]

Your destripe() ported need cullresize.dll same destripe in vs (.py), avs (.avsi) or don't need anymore??
https://github.com/YomikoR/VapourSynth-Destripe#related

*.mp4 guy's DeStripe (https://forum.doom9.org/showthread.php?t=154863&page=2#post1408276) has never required CullResize. The other Destripe script is a scaling script that uses CullResize and is used for fixing cross-conversion.

PS: there's an edit button that helps avoid posting back to back :p

[guest]

Hello again,

Can someone please tell me whats missing with this script :-

#Custom
LoadPlugin("%AVISYNTHPLUGINS%\mvtools\mvtools2.dll")
LoadPlugin("%AVISYNTHPLUGINS%\BM3D\x64\BM3DCPU_AVS.dll")
LoadPlugin("%AVISYNTHPLUGINS%\BM3D\x64\BM3D_VAggregate_AVS.dll")
LoadPlugin("%AVISYNTHPLUGINS%\dfttest\dfttest.dll")
LoadPlugin("%AVISYNTHPLUGINS%\RgTools\RgTools.dll")
Import("%AVISYNTHPLUGINS%\scripts\extras\ExTools52.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Sharpeners Pack v0.5.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\SMDegrain326d.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Utils-r41.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Zs_RF_Shared.avs")
video=SMDegrain(video,tr=4,thSAD=400,contrasharp=true,refinemotion=true,prefilter=5)

It has a error with Line 160 of SMDegrain3.2.6d.


But if I run this, there's no error :-

#Custom
LoadPlugin("%AVISYNTHPLUGINS%\mvtools\mvtools2.dll")
LoadPlugin("%AVISYNTHPLUGINS%\BM3D\x64\BM3DCUDA_AVS.dll")
LoadPlugin("%AVISYNTHPLUGINS%\BM3D\x64\BM3D_VAggregate_AVS.dll")
LoadPlugin("%AVISYNTHPLUGINS%\dfttest\dfttest.dll")
LoadPlugin("%AVISYNTHPLUGINS%\RgTools\RgTools.dll")
Import("%AVISYNTHPLUGINS%\scripts\extras\ExTools52.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Sharpeners Pack v0.5.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\SMDegrain326d.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Utils-r41.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Zs_RF_Shared.avs")
video=SMDegrain(video,tr=4,thSAD=400,contrasharp=true,refinemotion=true,prefilter=5)

BTW, I'm using either a 3950X or a 5950X.

:thanks:

[kedautinh12]

Hello again,

Can someone please tell me whats missing with this script :-



It has a error with Line 160 of SMDegrain3.2.6d.


But if I run this, there's no error :-



BTW, I'm using either a 3950X or a 5950X.

:thanks:

In BM3D_CPU don't have fast=true, extractor_exp=6, just deleted it when you change BM3D_CUDA to BM3D_CPU
https://github.com/Dogway/Avisynth-Scripts/blob/19f0b90838e0a7022fbf9c698c71ff201f7765bf/SMDegrain%20v.3.2.6d/SMDegrain%20v3.2.6d.avsi#L617

[guest]

In BM3D_CPU don't have fast=true, extractor_exp=6, just deleted it when you change BM3D_CUDA to BM3D_CPU
https://github.com/Dogway/Avisynth-Scripts/blob/19f0b90838e0a7022fbf9c698c71ff201f7765bf/SMDegrain%20v.3.2.6d/SMDegrain%20v3.2.6d.avsi#L617

You're awesome, I could make a "custom" SMDegrain with those deleted, and get the script to call the modded one.

I'll give it a shot, and let you know :)

Reply:- OK, that didn't work at first, but when I changed something else, it did, but it's SLOW !!!!

BM3D_CUDA(sigma=[s,ch?s/2:0,ch?s/2:0], radius=r, chroma=ch, fast=true, extractor_exp=6)
to
BM3D_CPU(sigma=[s,ch?s/2:0,ch?s/2:0], radius=r, chroma=ch)

Maybe that could be tweaked....

[kedautinh12]

Try this script i shared with BM3D_CPU & BM3D_CUDA. After benchmark between them
video=SMDegrain(video,tr=4,thSAD=400,contrasharp=true,refinemotion=true,prefilter=5)
Prefetch(16) #or try any number can make a good speed fps

[Boulder]

I personally wouldn't put too much effort on the prefiltering part. I use this kind of prefiltering method that was snatched from QTGMC (where it is used for prefiltering as well).

w = clip.Width()
h = clip.Height()
srchClip = clip.RemoveGrain(12,12).GaussResize(w,h, 0,0, w+0.0001,h+0.0001, p=2).MergeLuma(clip, 0.1)

[StainlessS]

Yo Doggy,
does anything in your Extools [or other] whotsits, measure distance of current pixel to nearest eg black[or white] pixel ? [maybe result pixel 0->255 for 8 bit]

Thanks in advance.

EDIT: Or suggestion. I'm not really expecting any +ve reply, but thought I'de ask.
Good Doggy :)

[Dogway]

Maybe you can try this. Binarize first, then expand (to calculate distance) and calculate output value.

ex_binarize(254)
aug = true # to see the mask

aug = aug ? "0.250001 *" : "range_max / "
ex2 = ex_expand(2)

Expr(last,ex_expand(1),ex2,ex2.ex_expand(1),Format("x y z a + + + "+aug),"")

[StainlessS]

Woah, I really was not expecting anything at all, I'll give it a go, thanks very much :)

[Dogway]

Just uploaded SMDegrain v3.2.7d, it includes a new function, ex_retinex(), its use is similar to ex_luma_rebuild() but more robust and a about half the speed although still fast.
I didn't like the implementations in VapourSynth or other sources so I directly ported from a paper and it works great. You can use it with SMDegrain argument "Str" and value 1.0, with value 0 it fallsback to frameprops to check if clip is TV levels and do a range conversion.
Also added a small optimization update to ex_minblur().

I also uploaded DeStripe() although either this or ex_retinex() make use of updates in ExTools that I still haven't uploaded, maybe this evening since I still want to do a quality check on ex_luts(), should be working fine but want to be sure.
I also ported from VapourSynth to ExTools Guided Blur and Anisotropic Guided Filtering, they are both slow. Will think for optimizations at another time.

[Julek]

Just uploaded SMDegrain v3.2.7d, it includes a new function, ex_retinex(), its use is similar to ex_luma_rebuild() but more robust and a about half the speed although still fast.

I get an error with ex_lutxyza, did you forget to add it or did I do something wrong?

[Dogway]

I also uploaded DeStripe() although either this or ex_retinex() make use of updates in ExTools that I still haven't uploaded, maybe this evening since I still want to do a quality check on ex_luts(), should be working fine but want to be sure.

I had a birthday tonight, will upload tomorrow ExTools update after some quality checks. I also think that ex_retinex() should better make use of AutoLevels instead of my runtime levels for better performance, will have a look tomorrow.

[Julek]

Oh, my bad.

[kedautinh12]

It's strange. Your ex_retinex use int sigma but MSRCP use float sigma[...] and your don't have upper_thr

[StainlessS]

Here something for Doggies Birthday[225 KB]:- https://www.mediafire.com/file/4fw6cgrzcds41dp/DogwayBirthday.7z/file

[Dogway]

Here something for Doggies Birthday[225 KB]:- https://www.mediafire.com/file/4fw6cgrzcds41dp/DogwayBirthday.7z/file

Haha thank you, did I misscommunicate? I didn't mean my birthday lol. Thanks anyway :rolleyes:

Just uploaded ExTools update. Didn't do a deep QC for ex_luts() today but should work fine for most things. If something odd arises I will fix on the go. I also had to update GradePack to allow floats in ex_levels() so ex_retinex() works fine.

Please test and if you see any issue let me know to fix it.

@kedautinh12: Just changed sigma to float. It wasn't an issue but I didn't port from anywhere. MSRCP is for RGB (chromaticity preservation), YUV doesn't need to preserve chroma as it is already detached from luma, one problem though is that YUV chroma is not entirely decorrelated, so one option I can add in the future is to preconvert to a better decorrelated color space. For SMDegrain prefiltering though this is not important.

@Julek: Test now with updated ExTools. Also if you let me know what type of Adaptive Threshold you use I can give it another stab. "Adaptive" is a generic family, some use integral other not. In your case it seemed to also use an edge mask. Here is a list of most kind of thresholds I could find.

Adaptive Mean
Adaptive Median
Adaptive Gaussian
Otsu Thresholding
Maximum Entropy Thresholding
Mixture Model Thresholding
Binary Clustering
Metric
Moment-Preserving Thresholding
Inner-class Variance
Pun Thresholding
Niblack Thresholding
Bernsen Thresholding
Abutaleb Thresholding
Sauvola Thresholding
Phansalkar Thresholding
PirahanSiah Adaptive Single thresholding based on PSNR ( www.pirahansiah.com )
Lloyd Thresholding
Ridler-Calvard Thresholding // Iterative Selection Thresholding
Johannsen Entropy Thresholding
Yen, Chang Thresholding
Sahoo, Wilkins, Yeager Thresholding // Renyi's Entropy Thresholding
Triangle Thresholding
Kittler-Illingworth Thresholding // Minimum Error Thresholding
Kapur, Sahoo, Wong Thresholding

[kedautinh12]

Thanks, can i useable ex_retinex(sigma=[50, 200, 350])??

[Dogway]

I haven't implemented custom sigmas, my reference paper did careful measurements to retrieve the optimal sigmas and weighted sums, so it is [1.5%, 5%, global mean] of width dimension and with weights [0.2, 0.1, 0.7]. 'sigma' arg is just a multiplier.
Do you have an example where this doesn't work fine?

[kedautinh12]

Oh, i try use your ex_retinex for port retinex_edgemask to avs+
https://github.com/Irrational-Encoding-Wizardry/kagefunc/blob/ab4f79a4d04a70bc83ef3e37d68c01fb4995a902/kagefunc.py#L217

[Dogway]

I don't have VS installed to compare results, so maybe that MSRCP call is not optimized enough, for example the sigmas there are fixed. I think you might be safe by replacing it entirely with ex_retinex(), if any play with the percentile value in my function to roughly match the dynamic range.

[tormento]

SMDegrain
Which filter do you suggest to lower HBD output from 16/12/10 bit SMDegrain to 8 bit? I am really not satisfied from the internal dithering used by x264 as it sometimes gives really visible banding.

I prefer modern filter ones. :)

[Dogway]

I'm yet to try fmtconv's Void & Cluster dithering, it's an ordered dither so maybe it compresses better than error diffusion, in my opinion it also distributes better so less banding.
Anyway my preference for the last years have been to (denoise), add grain and increase bitrate, I like texture in the images.

[guest]

I would like to ask a real dumb, noobie question about SMDegrain & pre-filter=4, KNLMeansCL, if I may.

How do you change the "strength" settings for KNLMeansCL ???, is it Line ***, or somewhere else ??

Example :-
#After_Prefetch_Denoise
LoadPlugin("%AVISYNTHPLUGINS%\KNLMeansCL\KNLMeansCL.dll")
video=KNLMeansCL(video,d=1, a=2, s=4, h=2, device_type="GPU", device_id=0)

And while I'm at it, does BM3D have optional settings within SMDegrain ??

TIA

[kedautinh12]

Here for document of KNLMeansCL
http://avisynth.nl/index.php/KNLMeansCL

BM3D don't have optional settings in SMDegrain. If you want change settings suitable for you. Can change here
https://github.com/Dogway/Avisynth-Scripts/blob/ec795d2ef5b515fa2e54427c4279541ee50fc438/SMDegrain%20v.3.2.7d/SMDegrain%20v3.2.7d.avsi#L159
Documents of BM3D:
http://avisynth.nl/index.php/BM3DCUDA

[Julek]

@Julek: Test now with updated ExTools. Also if you let me know what type of Adaptive Threshold you use I can give it another stab. "Adaptive" is a generic family, some use integral other not. In your case it seemed to also use an edge mask. Here is a list of most kind of thresholds I could find.

Adaptive Mean
Adaptive Median
Adaptive Gaussian
Otsu Thresholding
Maximum Entropy Thresholding
Mixture Model Thresholding
Binary Clustering
Metric
Moment-Preserving Thresholding
Inner-class Variance
Pun Thresholding
Niblack Thresholding
Bernsen Thresholding
Abutaleb Thresholding
Sauvola Thresholding
Phansalkar Thresholding
PirahanSiah Adaptive Single thresholding based on PSNR ( www.pirahansiah.com )
Lloyd Thresholding
Ridler-Calvard Thresholding // Iterative Selection Thresholding
Johannsen Entropy Thresholding
Yen, Chang Thresholding
Sahoo, Wilkins, Yeager Thresholding // Renyi's Entropy Thresholding
Triangle Thresholding
Kittler-Illingworth Thresholding // Minimum Error Thresholding
Kapur, Sahoo, Wong Thresholding

Here what I use:
https://docs.opencv.org/4.5.1/d7/d4d/tutorial_py_thresholding.html
cv.ADAPTIVE_THRESH_MEAN_C and cv.ADAPTIVE_THRESH_GAUSSIAN_C, using OpenCV + VS is already quite fast, so I don't know if via Expr would be faster.

[guest]

Here for document of KNLMeansCL
http://avisynth.nl/index.php/KNLMeansCL

BM3D don't have optional settings in SMDegrain. If you want change settings suitable for you. Can change here
https://github.com/Dogway/Avisynth-Scripts/blob/ec795d2ef5b515fa2e54427c4279541ee50fc438/SMDegrain%20v.3.2.7d/SMDegrain%20v3.2.7d.avsi#L159
Documents of BM3D:
http://avisynth.nl/index.php/BM3DCUDA

Thanks again, I thought it might be a "line" in SMDegrain somewhere :)

Not sure what's happened, but I am now getting x264 errors, and I can't seem to diagnose them :-

y4m [error]: bad sequence header magic
x264 [error]: could not open input file `-'

[kedautinh12]

Maybe your name of video file contain special characters or characters don't belong to English alphabet. Others your source video (L-SMASH) can't open that video format

[guest]

Maybe your name of video file contain special characters or characters don't belong to English alphabet. Others your source video (L-SMASH) can't open that video format

Yes, I wonder if it is the "name", thanks for the prompt.

Now that you've mentioned L-Smash....

I was going to post a comment on what ppl thought the best decoder was out of FFMS2 or L-Smash ???

What do you think is the "best" (non GPU based) ?

[kedautinh12]

Yes, I wonder if it is the "name", thanks for the prompt.

Now that you've mentioned L-Smash....

I was going to post a comment on what ppl thought the best decoder was out of FFMS2 or L-Smash ???

What do you think is the "best" (non GPU based) ?

Don't have best Video Source plugin, choose video source plugin base your video format

[tormento]

I'm yet to try fmtconv's Void & Cluster
Thanks, I will try it too.

It would be nice to have 2 parameters in SMDegrain where you can decide the internal engine bitdepth and the output one.

[Dogway]

I have always been reticent of adding too many options if they are not well justified, the reason, humans don't deal well when faced with too many options, they stress out and paralyze (FOBO (theurbanengine.com/blog//the-fear-of-better-options)).

FOBO (Fear of Better Options) is a subjective, first-person emotion of indecision, regret and frustration which is associated with being unable to identify a path of action due to an abundance of options.

I think that ConvertBits() is well and nice in its own, visible on the uppermost layer of the script so you can follow the bitdepth conversion flow.

[guest]

Maybe your name of video file contain special characters or characters don't belong to English alphabet. Others your source video (L-SMASH) can't open that video format

Well, I have tried a lot of different things to diagnose this, but it seems to be on only 4 servers that I use for the Distributed Encoding function of RipBot.

So they are simply connected via a LAN, and don't need the complete program to encode (very clever, actually), and yet other pc's on the LAN don't have the problem,

So I think my only recourse is a fresh setup of Windows.

Update:- It turned out that I needed to update these:-VC_redist.x86.exe & VC_redist.x64.exe

There must be some subtle change in Dog's ExTools & SMDegrain.

[tormento]

I think that ConvertBits() is well and nice in its own, visible on the uppermost layer of the script so you can follow the bitdepth conversion flow.
I have tried Void & Cluster and I can't really see lot of improvements over x264 internal Sierra dithering. Perhaps I need to touch knobs that I don't really know, such as pattern size and so.

[Dogway]

Well, there's another option, to use SmoothGrad() it uses Dither Tools, maybe there's an equivalent in fmtconv but haven't checked. In any case I have plans of porting it to ExTools from VS ports in a new pack called yugefunc.avsi

[StainlessS]

@Pauly Dunne [or anyone else]

Here is abbodi1406 vcredist AIO (All In One) version 52, 1st Aug 2021.
[27MB]:- https://github.com/abbodi1406/vcredist/releases

All runtimes from 2005 to current for x86 and x64 combined, all un-necessary stuff stripped out and much smaller than M$ download of all separately.

Good idea to keep the link, updated fairly frequently [about once per month, when M$ makes updates].

[guest]

@Pauly Dunne [or anyone else]

Here is abbodi1406 vcredist AIO (All In One) version 52, 1st Aug 2021.
[27MB]:- https://github.com/abbodi1406/vcredist/releases

All runtimes from 2005 to current for x86 and x64 combined, all un-necessary stuff stripped out and much smaller than M$ download of all separately.

Good idea to keep the link, updated fairly frequently [about once per month, when M$ makes updates].

Nice, StainlessS, can't remember where I found mine, but it's the latest build 14.30.30423.0.

Also need to mention I think it's wise to install both x86 & x64, as I had a fairly recent build of x64, installed, but rather old x86, but after installing, everything works again :)

[real.finder]

Well, there's another option, to use SmoothGrad() it uses Dither Tools, maybe there's an equivalent in fmtconv but haven't checked. In any case I have plans of porting it to ExTools from VS ports in a new pack called yugefunc.avsi

SmoothGrad still missing, along with Dither_box_filter16 and dither_bilateral16 https://forum.doom9.org/showpost.php?p=1943559&postcount=1131

[Dogway]

Dither_box_filter16 is ex_boxblur(mode="mean"), for Dither_bilateral16() we have ex_bilateral() or vsTBilateral(), I still need to port LimitFilter (https://github.com/HomeOfVapourSynthEvolution/mvsfunc/blob/7948c8be129bc9cb282cf24e25b3c4b77328a9e0/mvsfunc.py#L1776)and SmoothGrad (https://github.com/WolframRhodium/muvsfunc/blob/80e99100c78b96b1c925a5989259ef22c1bc6173/muvsfunc.py#L1899).

[kedautinh12]

Here for new ver of mvsfunc. If you want port something from vs. Check here:
https://github.com/AmusementClub/mvsfunc

[real.finder]

Dither_box_filter16 is ex_boxblur(mode="mean"), for Dither_bilateral16() we have ex_bilateral() or vsTBilateral(), I still need to port LimitFilter (https://github.com/HomeOfVapourSynthEvolution/mvsfunc/blob/7948c8be129bc9cb282cf24e25b3c4b77328a9e0/mvsfunc.py#L1776)and SmoothGrad (https://github.com/WolframRhodium/muvsfunc/blob/80e99100c78b96b1c925a5989259ef22c1bc6173/muvsfunc.py#L1899).

ex_bilateral give same output as Dither_bilateral16?

LimitFilter = https://github.com/realfinder/AVS-Stuff/blob/b71a487e3f9619a5a23162215412abcad024aa56/avs%202.5%20and%20up/Zs_RF_Shared.avsi#L476

[real.finder]

Dither_box_filter16 is ex_boxblur(mode="mean")

#ex_boxblur(5,mode="mean")
Dither_convert_8_to_16().Dither_box_filter16(5).DitherPost()

but I can see that they are not similar

[Dogway]

#ex_boxblur(5,mode="mean")
Dither_convert_8_to_16().Dither_box_filter16(5).DitherPost()

but I can see that they are not similar

They are the same, the only difference is that Dither_box_filter16 is frequency domain based so it's faster for higher sigma, when above 5 in ex_boxblur().
a=Dither_convert_8_to_16().Dither_box_filter16(5).DitherPost()
ex_boxblur(4,mode="mean")
ex_makediff(a, aug=true, UV=128)

So if the question is if we need a DFT box filter, probably the answer is yes, same with binomial blur. Of course with HBD support and asymmetric blur option. I would also vouch for porting CTM (Constant Time Median) from VapourSynth, so doing a 5x5 or 7x7 median isn't as slow as it is now.

ex_bilateral() gives same output as vsTBilateral(), speed is similar as well. Dither_bilateral16() employs shortcuts to speed up the process but in my opinion the output doesn't look nice.

Excerpt from docs.
"Actually, this implementation is an approximation of a bilateral filter, like many other image processing tools do. The spatial weighting is not a gaussian curve but a box, and the value weighting is a trapezoid."

Thanks for LimitFilter link, I will have a look when I got time, I've been busy porting *.mp4 sharpeners

[real.finder]

a=Dither_convert_8_to_16().Dither_box_filter16(5).DitherPost()
ex_boxblur(4,mode="mean")
ex_makediff(a, aug=true, UV=128)

so Dither_box_filter16(x) = ex_boxblur(x-1, mode="mean")

CTM (Constant Time Median) from VapourSynth

https://github.com/pinterf/MedianBlur2

Dither_bilateral16() employs shortcuts to speed up the process but in my opinion the output doesn't look nice.

but there are people who like the Dither_bilateral16 more than others

[Dogway]

MedianBlur2 didn't perform well on my benchmarks, something is going on there, occasionally it would perform "fine" but must of the time it crawls so I don't know.

I will try to post comparison shots for ex_bilateral() and Dither_bilateral16() when I got some time.

[Dogway]

Here's the comparison (brighten to see better):

Converttostacked().Dither_bilateral16(radius=2,thr=10,flat=1.0,u=1,v=1).Convertfromstacked()
https://i.imgur.com/hoy2mTk.png

ex_bilateral(1)
https://i.imgur.com/qHznfYB.png

As you can see Dither_bilateral16() output is very dirty/grainy, and artifacts similar to combing.



By the way, updated SharpenersPack with new sharpeners; SSSharp, ReCon, blah, MedSharp, ex_unsharp and SharpenComplex2

[real.finder]

Here's the comparison (brighten to see better):

Converttostacked().Dither_bilateral16(radius=2,thr=10,flat=1.0,u=1,v=1).Convertfromstacked()
https://i.imgur.com/hoy2mTk.png

ex_bilateral(1)
https://i.imgur.com/qHznfYB.png

As you can see Dither_bilateral16() output is very dirty/grainy, and artifacts similar to combing.



By the way, updated SharpenersPack with new sharpeners; SSSharp, ReCon, blah, MedSharp, ex_unsharp and SharpenComplex2

I think it depends on the settings and the source

here a friend words

https://i.postimg.cc/Kc5D2Zh0/1.png (https://postimages.org/)

https://i.postimg.cc/wTTcxS6g/2.png (https://postimages.org/)

[guest]

Well, I have tried a lot of different things to diagnose this, but it seems to be on only 4 servers that I use for the Distributed Encoding function of RipBot.

So they are simply connected via a LAN, and don't need the complete program to encode (very clever, actually), and yet other pc's on the LAN don't have the problem,

So I think my only recourse is a fresh setup of Windows.

Update:- It turned out that I needed to update these:-VC_redist.x86.exe & VC_redist.x64.exe

There must be some subtle change in Dog's ExTools & SMDegrain.

I've finally figured out what the problem is....

Even tho it was good to update VC, it's an OpenCL problem when using KNLMeansCL, as a pre-filter in Doggy's SMDegrain !!!

The pc's that have this error haven't got a compatible GPU, and I'm not sure if you can get a "driver" for older Xeon Intel CPU's for OpenCL Support.

[kedautinh12]

Wait a minute, i remember you said you had Ryzen cpu??

[guest]

Wait a minute, i remember you said you had Ryzen cpu??

Good memory, I actually have 2 :)

I also have a few I7 Intels, some E5 Xeons, And a LOT of older X series Xeons.

And the older Xeons have no "proper" Gpu's....

Its a RipBot264 Distributed Encoding set up...

[kedautinh12]

Good memory, I actually have 2 :)

I also have a few I7 Intels, some E5 Xeons, And a LOT of older X series Xeons.

And the older Xeons have no "proper" Gpu's....

Its a RipBot264 Distributed Encoding set up...

Rich oldman. When you want discard any old amd ryzen 5000 series, can you give me :D :D :D

[Emulgator]

Be kind, ask for grannys. These are good too ;-)

[guest]

You welcome

;)

I'll see you @ the other Forum.

[Dogway]

ExTools (https://github.com/Dogway/Avisynth-Scripts/blob/master/ExTools.avsi) updated to v5.4
-ex_vinverse() - small optimization
-ex_blur() - 'bifit' arg to toggle between gaussian or binomial distribution
-ex_guidedblur(), ex_ANguidedblur() - moved to yugefunc.avsi
-ex_GaussianBlur() - New. Optimized true gaussian blur approximation for high sigma
-ex_luts() - bugfix when using two clips. 'exprf' arg to append code to the end.
-ex_shape() - add anisotropy for shapes (except 'cross')
-sel_med() - more medians, fill up to n=49

Just updated ExTools to v5.4, required for my recent updates (from today updates, and before for SharpenersPack). This feels like the last version although I will fix any reported bugs or add things if it feels convenient, but my update rate on ExTools will be much slower from now on.
I also uploaded today v1.0 of SharpenersPack (https://github.com/Dogway/Avisynth-Scripts/blob/master/SharpenersPack.avsi), because it feels very solid. There are possible additions like Oyster, qUINT Sharp, NLM Sharp and other kind of sharpeners, but for now it will stay like this.
Also updated SMDegrain (https://github.com/Dogway/Avisynth-Scripts/blob/master/SMDegrain%20v.3.2.8d/SMDegrain%20v3.2.8d.avsi)to refine ex_retinex() with the new ex_GaussianBlur() function.

Now I will focus on yugefunc (https://github.com/Dogway/Avisynth-Scripts/blob/master/yugefunc.avsi)and port worthy functions from VapourSynth, I added a list in the header for the filters I wish to port.
I will also have a look at QTGMC to optimize it more, port Didée's LTSMC, remake ex_vibrance() and whatever arises.

[guest]

ExTools (https://github.com/Dogway/Avisynth-Scripts/blob/master/ExTools.avsi) updated to v5.4
-ex_vinverse() - small optimization
-ex_blur() - 'bifit' arg to toggle between gaussian or binomial distribution
-ex_guidedblur(), ex_ANguidedblur() - moved to yugefunc.avsi
-ex_GaussianBlur() - New. Optimized true gaussian blur approximation for high sigma
-ex_luts() - bugfix when using two clips. 'exprf' arg to append code to the end.
-ex_shape() - add anisotropy for shapes (except 'cross')
-sel_med() - more medians, fill up to n=49

Just updated ExTools to v5.4, required for my recent updates (from today updates, and before for SharpenersPack). This feels like the last version although I will fix any reported bugs or add things if it feels convenient, but my update rate on ExTools will be much slower from now on.
I also uploaded today v1.0 of SharpenersPack (https://github.com/Dogway/Avisynth-Scripts/blob/master/SharpenersPack.avsi), because it feels very solid. There are possible additions like Oyster, qUINT Sharp, NLM Sharp and other kind of sharpeners, but for now it will stay like this.
Also updated SMDegrain (https://github.com/Dogway/Avisynth-Scripts/blob/master/SMDegrain%20v.3.2.8d/SMDegrain%20v3.2.8d.avsi)to refine ex_retinex() with the new ex_GaussianBlur() function.

Now I will focus on yugefunc (https://github.com/Dogway/Avisynth-Scripts/blob/master/yugefunc.avsi)and port worthy functions from VapourSynth, I added a list in the header for the filters I wish to port.
I will also have a look at QTGMC to optimize it more, port Didée's LTSMC, remake ex_vibrance() and whatever arises.

Busy, busy, busy...I'm having a hard time just keeping up :)

:thanks:

Edit :- Unfortunately, since changing up to SMDegrain 3.2.8, I'm getting another Line 161 error...go back to 3.2.7, it's good :), when using BM3D in CPU mode.

On further investigation, the difference was :-

function ex_BM3D(clip a, float "sigma", int "radius", int "UV") {

rgb = isRGB(a)
isy = isy(a)
bi = BitsPerComponent(a)

s = Default(sigma, 3)
r = Default(radius, 1)
UV = Default(UV, rgb ? 3 : 1)

ch = !(isy || UV!=3)
ch ? ConverttoYUV444(a,chromaresample="spline16") : a
ConvertBits(bits=32, fulls=fs)

BM3D_CPU(sigma=[s,ch?s/2:0,ch?s/2:0], radius=r, chroma=ch)
BM3D_VAggregate(radius=r)

if (bi < 32) {
ConvertBits(bits=16, dither=-1, fulls=fs)
ConvertBits(bits=bi, dither= 1, fulls=fs)
}
ch ? MatchColorFormat(a) : mergechroma(a) }

full=fs was the only change I could see, so whatever that's for, didn't work for me. :(

Also note other change in "red" (thanks to kedautinh12)

[tormento]

Unfortunately, since changing up to SMDegrain 3.2.8, I'm getting another Line 161 error...go back to 3.2.7, it's good :), when using BM3D in CPU mode.
Updated to last BM3D of a few days ago?

[tormento]

I will also have a look at QTGMC to optimize it more, port Didée's LTSMC, remake ex_vibrance() and whatever arises.
Is Zs_RF_Shared still required to use SMDegrain?

[guest]

Updated to last BM3D of a few days ago?

I only have a problem when using it in CPU mode..

[Dogway]

@Pauly Dunne: Thanks for the report! I updated SMDegrain, last minute changes are never good. Also added CPU mode but it's not callable from SMDegrain args, just change to CUDA=false in line 159.

@tormento: I think it's needed only for sh_GetUserGlobalIMTint(), but when I used it it didn't report my threads. I still don't know how AviSynth doesn't have a thread reporting internal function...

[real.finder]

@tormento: I think it's needed only for sh_GetUserGlobalIMTint(), but when I used it it didn't report my threads. I still don't know how AviSynth doesn't have a thread reporting internal function...

there are https://forum.doom9.org/showthread.php?p=1925253#post1925253 but seems no one use it

[Dogway]

Can avstp report cores/thread as string?

Here an image for ex_shapes(), still need to work out cross anisotropy (EDIT: fixed):
https://i.imgur.com/VAu8ahP.gif

[real.finder]

Can avstp report cores/thread as string?

don't think so, I didn't mean avstp but https://forum.doom9.org/showthread.php?p=1777021#post1777021 and https://forum.doom9.org/showthread.php?p=1778346#post1778346

[Dogway]

Yes, the MT resizers. I used them some time ago, then I benchmarked recently and wasn't as fast as internal. Maybe you need several resizer calls to see the improvement, I will test that but IMO these kind of things should be dealt directly in AVS+, I for one am aware of the issue and try to merge redundant Expr() calls into one to reduce the call penalty (used registers).

By the way, big update day. SMDegrain, QTGMC, ExTools, GradePack, etc
SMDegrain: reworked ex_retinex() to don't cause flicker in new 'lvl' mode 2 while keeping performance, 'lvl' 1 uses fixed levels (Default)
ExTools: basically adding 'tv_in', 'tv_out' args to ex_bs() so you can convert values from PC levels to TV levels and opposite.
QTGMC: Performance update, some cleanup, and a few small bugs. Still needs 32-bit support.
GradePack: reworked chroma in ex_contrast() and ex_levels() to make more sense. Actually ex_levels() reaches parity with Levels() regarding chroma. Also added more dither modes to ex_posterize() by using fmtc_bitdepth(). Still need to update ex_vibrance().
Other: Small updates to ResizersPack, DeStripe, FastLineDarkenMOD, LSFMod, FrameRateConverter and GrainFactory3mod.

Please report if you find any issue.

[guest]

Yes, the MT resizers. I used them some time ago, then I benchmarked recently and wasn't as fast as internal. Maybe you need several resizer calls to see the improvement, I will test that but IMO these kind of things should be dealt directly in AVS+, I for one am aware of the issue and try to merge redundant Expr() calls into one to reduce the call penalty (used registers).

By the way, big update day. SMDegrain, QTGMC, ExTools, GradePack, etc
SMDegrain: reworked ex_retinex() to don't cause flicker in new 'lvl' mode 2 while keeping performance, 'lvl' 1 uses fixed levels (Default)
ExTools: basically adding 'tv_in', 'tv_out' args to ex_bs() so you can convert values from PC levels to TV levels and opposite.
QTGMC: Performance update, some cleanup, and a few small bugs. Still needs 32-bit support.
GradePack: reworked chroma in ex_contrast() and ex_levels() to make more sense. Actually ex_levels() reaches parity with Levels() regarding chroma. Also added more dither modes to ex_posterize() by using fmtc_bitdepth(). Still need to update ex_vibrance().
Other: Small updates to ResizersPack, DeStripe, FastLineDarkenMOD, LSFMod, FrameRateConverter and GrainFactory3mod.

Please report if you find any issue.

I thought you'd finished modding these....:eek:

No problems for me :)

[Julek]

I don't know if this has been ported to AVS yet, but chroma recontructing would be a good function for you.
And since the AVS's Expr is more complete than the VS's Expr you might get an even better result.

Func:
https://github.com/Irrational-Encoding-Wizardry/lvsfunc/blob/master/lvsfunc/recon.py#L70-L119
https://en.wikipedia.org/wiki/Simple_linear_regression

Usage:
https://github.com/Ichunjo/encode-scripts/blob/b016e0b1e64e627051086a6e54555e9b14e7a11b/Raws-Maji/Lapis%20ReLiGHTs%20%5BBD%5D/lapis_bdmenu03.py#L72-L93
https://github.com/Ichunjo/encode-scripts/blob/b016e0b1e64e627051086a6e54555e9b14e7a11b/Other%20groups/MTBB/Sirius%20%5BBD%5D/sirius_03.py#L182-L207
https://github.com/Ichunjo/encode-scripts/blob/b016e0b1e64e627051086a6e54555e9b14e7a11b/Raws-Maji/Rifle%20is%20Beautiful%20%5BBD%5D/rifle_bdmenu_02.py#L68-L90

Result: https://slow.pics/c/MfqBYVId

[Dogway]

I thought you'd finished modding these....:eek:

ex_retinex() always had problems with the autolevels, so it was prone to change. Yesterday I also tested gamma aware blurring for retinex and results were worse even after normalization, so I'm always testing how to improve things or make them faster. The core of SMDegrain though is pretty much done with the current concept. MatteCrop needs an update for the lookback loop, I will replace it with frameProps but didn't have time yet to implement. GrainFactory3mod is slow, I want to replace it with something new. It's a bit of a cleanup before I move to yugefunc.


@Julek: Looks good! anything that is math clean and fast is welcome. I will have a look at linear regression and see how it fits although the chroma reconstruction function shown includes some over the top filtering with MaskedDHA, QTGMC, etc. I already had a chroma reconstruction (https://github.com/Dogway/Avisynth-Scripts/blob/205307cba48b581634b91875496aebf7906981ae/TransformsPack.v1.0.RC18.avsi#L376) function (https://github.com/Dogway/Avisynth-Scripts/blob/205307cba48b581634b91875496aebf7906981ae/MIX%20mods/ChromaReconstructor%20v3mod.avsi#L92)ported from feisty2.
By the way yesterday I ported adaptive threshold successfully, I'm still on optimization stage. Do you have performance numbers for 1080p@16-bit (without bitdepth down) and CPU? I'm managing 3 parameters, threshold, blur scale, and integral size.

[anton_foy]

Tried using ex_median() and I get error: Script error: Expr: Failed to convert 'max to float' (extools.avsi, line 2511)

Am I missing something?

[Dogway]

"DGM0" ? Works fine here. Are you in avs+ 3.7.1?

[anton_foy]

"DGM0" ? Works fine here. Are you in avs+ 3.7.1?

I sure hope so, because it gave me another error message "need to upgrade to avisynth 3.7.1+" before so I upgraded and now this error message turned up (I replaced the old avisynth.dll with the new x86 3.7.1 one in sysWOW64 and system32 folder)

[Dogway]

In system32 goes the one with CRC32: D12B9040 and in SysWOW64 the one: 4B48E737 (size is bigger here)

EDIT: you need version test9, check here (https://forum.doom9.org/showthread.php?p=1946928#post1946928).

[anton_foy]

In system32 goes the one with CRC32: D12B9040 and in SysWOW64 the one: 4B48E737 (size is bigger here)

Sorry for the stupid off topic questions. I use 32-bit Avisynth+.
Which ones are CRC32: D12B9040 and 4B48E737?
Two different avisynth.dll's?

[Dogway]

Yes, x64 version goes to system32 and x86 goes to sysWOW64 if I reckon correctly. Check the link in my edit above. Then with avsmeter you can check your install is fine: "avsmeter64 avsinfo -lf"

@Julek: ex_athres(th=40,sigma=6) running here at 194fps for my CPU with Prefetch(8). It can be sped up slightly but I need this (https://forum.doom9.org/showthread.php?p=1951860#post1951860) avs+ bug fixed.

[anton_foy]

Yes, x64 version goes to system32 and x86 goes to sysWOW64 if I reckon correctly. Check the link in my edit above. Then with avsmeter you can check your install is fine: "avsmeter64 avsinfo -lf"


:thanks: it is working now!

[Julek]

@Julek: ex_athres(th=40,sigma=6) running here at 194fps for my CPU with Prefetch(8). It can be sped up slightly but I need this (https://forum.doom9.org/showthread.php?p=1951860#post1951860) avs+ bug fixed.

3001 frames 1080p (AMD 3700X)
src: Time elapsed: 0:06.013 - 499.10511998696642876894 FPS
VS + OpenCV adaptiveThresh (luma only) : Time elapsed: 0:11.184 - 268.33492243462092119444 FPS

I can only test in 8 bits, because that's how VS+CV works

[Dogway]

Then more or less the same if it was processed in 16-bit, will upload tomorrow if possible with multi-otsu or adaptive histogram.

[guest]

In system32 goes the one with CRC32: D12B9040 and in SysWOW64 the one: 4B48E737 (size is bigger here)

EDIT: you need version test9, check here (https://forum.doom9.org/showthread.php?p=1946928#post1946928).

Hey Dogway,

After clicking on the link, I found test 11 & test 12 of 3.7.1, so naturally I grabbed them, unfortunately both test 11 & 12 give me a Line 161 error in SMDegrain 3.2.9d :(

So I had to return to test 9 :) or :(

[Dogway]

Yeah, it's because this commit (https://github.com/AviSynth/AviSynthPlus/commit/f89f473ffb62f3a96457e085aad305318b37e7a6) yesterday, guess my trick doesn't work now, will have to ask WolframRhodium to make sigma arg array type. I will update it.

@Julek: Just uploaded ex_athres(). It's not mathematically exact to the OpenCV implementation because I didn't check its source code but comparing with your test clips and other samples I found it works wonders. Now it looks to me more like an edge detection filter than a threshold per se. You need AVS+ 3.7.1 test12 released yesterday for the integral optimizations. It's basically a DoBtG (Difference of Box to Gaussian) but the integral helps to clean the output a bit more.

[real.finder]

Yeah, it's because this commit (https://github.com/AviSynth/AviSynthPlus/commit/f89f473ffb62f3a96457e085aad305318b37e7a6) yesterday, guess my trick doesn't work now, will have to ask WolframRhodium to make sigma arg array type. I will update it.


https://github.com/WolframRhodium/VapourSynth-BM3DCUDA/actions/runs/1215772307

[guest]

Yeah, it's because this commit (https://github.com/AviSynth/AviSynthPlus/commit/f89f473ffb62f3a96457e085aad305318b37e7a6) yesterday, guess my trick doesn't work now, will have to ask WolframRhodium to make sigma arg array type. I will update it.

Hey Dogway,

Have been updating things again, with Extools 5.6, etc. and I grabbed this :-

https://github.com/WolframRhodium/Va...uns/1215772307 (thanks real.finder)

Wolfram changed quite a bit in the latest build :)

re-addressed my scripts, and it now seems to be working with AVS test 12 :)

[Julek]

@Julek: Just uploaded ex_athres()

It's not working here.
Expr: Failed to convert 'B^' to float
Same thing with 3.7.1 test12

[Dogway]

Are you on linux? It certainly looks like it's not returning the next string line correctly, this was implemented a few test versions earlier but at least it works for me on Windows.

EDIT: Can you run the next?
Expr(last," x X^
range_max X - abs","")

The line carriage in ExTools should be Line Feed -> "LF"

Test other examples like ex_median(mode="median5")

[Julek]

I am on windows 10, the problem was the line break, I edited it to one line and it worked.
I'm not very familiar with AVS and may not have installed test12 correctly.
I will do more tests tonight.

[Julek]

I did some tests now and the result is awesome, maybe even better than OpenCV for my use,
and the best thing is that it gets this result without the luma equalization that I used with OpenCV,
so the speed is much higher if you take that into account, great job.

Now I have to wait for Akarin to update his plugin, because I cannot use B^ in VS Expr :(
https://github.com/AkarinVS/vapoursynth-plugin

#EDIT: I think I learned how to use Akarin's "var!" and "var@", it can replace the "^".

[Dogway]

Good to hear. I don't know about the luma equalization. If you installed avs+ test 12 correctly you shouldn't have issues with the line carriage.
By the way my gaussian blur values are true gaussian sigma approximations not binomial so it might differ depending on what you use on VS.

[tormento]

Good to hear.
I am having problems with the "cosmetical enhanced" version of ExTools 5.6. The same version old one works ok.

Error:

(D:/Programmi/Media/AviSynth+/plugins64/ExTools-5.6new~Dogway.avsi, line 514, column 14)

[Dogway]

Thank you, fixed now. I always get the order wrong of ">="

[wonkey_monkey]

Thank you, fixed now. I always get the order wrong of ">="

The crocodile ( > ) always wants to eat the bigger number: 8 > 3

https://img.ifunny.co/images/a0775459ee5b597246cf16c5f1225356a3e7c1fb6347f2179d7e0969e717de72_1.webp

[Dogway]

Hehe, it's actually whether >= or =>. In language normally is said "equal or better/greater"

[kedautinh12]

Hehe, it's actually whether >= or =>. In language normally is said "equal or better/greater"

In my country use "=>" mean arrow :D

[tormento]

Hehe, it's actually whether >= or =>. In language normally is said "equal or better/greater"
Can I suggest you to use Notepad3 (https://github.com/rizonesoft/Notepad3) with a glyph enabled font, such as Cascadia or Iosevka?

If you get the correct order, it will transform the >= to ≥ but only visually.

There is AVS syntax highlight already present.

[Dogway]

Can I suggest you to use Notepad3 (https://github.com/rizonesoft/Notepad3) with a glyph enabled font, such as Cascadia or Iosevka?

Oh my gosh, thank you! I already use Notepad2, didn't have a clue about a third version, thought it was abandoned.

It might be simple for some who work on full fledged IDEs, but this is my most heavily used program on the PC. Hopefully it has improved regex and fixed crashes with "Mark Occurrences"

[tormento]

Oh my gosh, thank you!
You are the most welcome.

I warmly suggest to join the GitHub community. The AVS syntax needs a refresh.

this is my most heavily used program on the PC

Here too. It had a Notepad debug integration, so that every program calling for Notepad would have used Notepad 3 but the latest builds of Windows removed that feature.

[StainlessS]

NotePad2[Florian Balmer] and NotePad3[Derick Payne], seem to be from different authors.
Also there is at least one more by same name (NotePad2) on sourceforge by Ikiki.

NotePad2[Florian Balmer]:- https://www.flos-freeware.ch/
NotePad3[Derick Payne]:- https://github.com/rizonesoft/Notepad3/releases
NotePad2[Ikiki]:- https://sourceforge.net/projects/notepad2/

EDIT: There dont seem to be a NotePad4, except on Android.

[Dogway]

I was using the notepad2-mod from XhmikosR here (https://github.com/XhmikosR/notepad2-mod/releases/tag/4.2.25.870). From 2013, guess it's time to update lol

[guest]

Wow, it's hard to keep up with all the mod's & revisions you're doing, "Dog"...I only updated some of these this morning... :)

[Dogway]

Most of the things are finished by now, just little things here and there. QTGMC is fine but can be improved further (on the long run) after I run some Zopti tests, same with SMDegrain, but the logic here is finished. I also plan to implement a temporal limiting mode to LSFMod to propagate into SMDegrain.
TransformsPack is in WIP, but ConvertFormat() works, I just left some chroma issue unfixed a few months ago and of course chroma placement which still haunts me.
I'm adding a few more sharpeners to SharpenersPack and finishing DeblockPack to upload soon.

I was having some issues matching fft3dfilter with FFT3DGPU with large sigma, I saw a post of someone matching both but I forgot where...

[kedautinh12]

About deblocking, i seen SmoothD2 and SmoothD2c still good but that support only yv12 and don't have HBD

[Dogway]

I include SmoothD2c but ultimately depends on SmoothD2 which is a plugin, so we are limited by that. I'm also porting SmoothDeblock, which is a bit of a nightmare, but I will go little by little. These kind of projects help me see what I'm missing in ExTools so I include them on the go to be able to port them easier.
I already ported Deblock_QED() gaining 29% of performance.

[kedautinh12]

About Deblock_QED(), autodeblock had this function and have more options to do: weakdeblock, mediumdeblock, strongdeblock, autodeblock
https://github.com/realfinder/AVS-Stuff/blob/Community/avs%202.5%20and%20up/autodeblock2.avsi

[Dogway]

Looks a bit over-the-top, but I might include it at some point given it gives good results or I can optimize it more.
This is the list of deblocks, those marked as Script are subject to be included.

I'm also including some Didée's suggestions for MC deblocking like MCSoothe, SoothCL, etc, based on FFT3D, which I try to port to FFT3DGPU. Once this is done I upload while SmoothDeblock still in WIP.
### Included: ###
### Deblock_QED (+Deblock plugin) ###
### SmoothD2c (+SmoothD2 plugin) ###
### MSUDeCor (+MSU_Smart_Deblocking plugin) ###
### SmoothDeblock (Script) ###
### Other (not included): ###
### BlindPP("xxxx")(DCDecode) ###
### Deblock(DGDecode,MVTools) ###
### DeblockM ###
### vsDeblockPP7 ###
### MSUDeblock ###
### MSUSmartDeblocking ###
### SmoothD2 ###
### Unblock ###
### BlockTerminator (Script) ###
### BlockKiller (Script) ###
### FunkyDeblock (Script) ###

[kedautinh12]

I seen MSU filter but it's VirtualDub plugin and can't autoload in avs+

Edit: unblock only support avs x86. Where can i get deblockM??

[Dogway]

I think he meant MDeblock here (https://forum.doom9.org/showthread.php?p=913365). Yes, you need to manually load the MSU filter.
I think the best Deblock_QED contender might be SmoothDeblock but it uses old procedures for loop, arrays, etc, and while it might work fine today I want to refactor it entirely to modern syntax, so that will take me some time.

[kedautinh12]

I think he meant MDeblock here (https://forum.doom9.org/showthread.php?p=913365). Yes, you need to manually load the MSU filter.
I think the best Deblock_QED contender might be SmoothDeblock but it uses old procedures for loop, arrays, etc, and while it might work fine today I want to refactor it entirely to modern syntax, so that will take me some time.

Link dead and i got MDeblock but it's still only support x86 ver

[guest]

Hi Dogway,

Grabbed Sharpeners Pack v1.5 (19-09-2021), and I am now having an error @ Line 556 Col 8, if !(MC && isg) {

no issues with previous build.

[tormento]

Dogway's Filters Packs

Script error: expected `('
(D:/Programmi/Media/AviSynth+/plugins64/SharpenersPack-1.5~Dogway.avsi, line 556, column 8)

Edit: so fast you were. Working :)

[Dogway]

Yes, sorry I seem to make the stupidest mistakes, fixed now.

By the way, I'm currently porting Plum, so I will upload soon.
Also porting Oyster to DeblockPack, but lack of features in AVS BM3D makes it difficult if not impossible to replicate vs version.

[kedautinh12]

Yes, sorry I seem to make the stupidest mistakes, fixed now.

By the way, I'm currently porting Plum, so I will upload soon.
Also porting Oyster to DeblockPack, but lack of features in AVS BM3D makes it difficult if not impossible to replicate vs version.

Yeah WolframRhodium only fork a part of vs-BM3D of HolyWu

[Dogway]

Finally uploaded DeblockPack, a nightmare. Except for Deblock_QED() I wouldn't use the rest yet, specially SmoothDeblock() and Oyster. Oyster should be "working" (aside from the missing settings in BM3D) but it actually hangs avspmod, probably because of the absurd values...

[anton_foy]

I tried your version of Deblock_QED and got:

Script error: ex_lutspa does not have a named argument "expr" DeblockPack.avsi, line 113
Am I missing something?

[Dogway]

Update ExTools I guess.

[anton_foy]

Update ExTools I guess.

Sorry yes that fixed it, thank you.

[Dogway]

By the way, uploaded FrameRateConverter 2.0, rebased to v2.0 beta 11. Works great, possibly faster than vanilla in 8-bits, and nearly as fast in 16-bit. Updated benchs on OP.
I also observed that blending is reduced when processing in HBD.

[WolframRhodium]

Yeah WolframRhodium only fork a part of vs-BM3D of HolyWu

True. And the denoising strength of the two filters is different. I do not recommend replacing VS-BM3D with the CUDA version without careful tuning at this moment.

[kedautinh12]

@Dogway Vapoursynth now support back win7. You can compare result between vs and avs+ when you want port vs filters or scripts to avs+
https://forum.doom9.org/showthread.php?p=1952807#post1952807

[real.finder]

@Dogway Vapoursynth now support back win7. You can compare result between vs and avs+ when you want port vs filters or scripts to avs+
https://forum.doom9.org/showthread.php?p=1952807#post1952807

when I use vs for compare I use https://forum.doom9.org/showthread.php?t=175529

[Dogway]

@Dogway Vapoursynth now support back win7. You can compare result between vs and avs+ when you want port vs filters or scripts to avs+
https://forum.doom9.org/showthread.php?p=1952807#post1952807

Thanks, yes I have that pending. Actually I have many things pending (vs, notepad3, PC cleanup, emulators setup), but it will come useful for yugefunc, the problem is my computer is full to the brim, I need to delete things, and pytorch and these things take a good amount of space. So when I get time to do cleanup and finish the main avs+ ports I want to try it.
I'm interested on using vs for the neural network filters and enhance some old photos.

[Dogway]

I'm finishing porting Plum. Yeah, I'm having many issues due to memory overflow. Currently using a 1080p source.
nnedi3
Requested buffer size of 6417285120 is too large

nnedi3
Error while allocating planar dstPF

KNLMeansCL: fatal error!
(clCreateContext: CL_OUT_OF_RESOURCES)
(New File (30), line 93)

I don't know who to blame here, the plugins, avisynth... in any case I think overall the framework should be updated to deal fine with 8K clips, not necessarily for 8K sources but also ss 4K or 2K sources. I have 32Gb of RAM and 8Gb of VRAM.


Also having issues with, returns Undefined() when further processing, ok as final call:
Expr(clp, SelectEvery(src, {radius} * 2 + 1, {i}), ...(x13), "x y max z max a max b max c max d max e max f max g max h max i max j max")




EDIT: Uploaded Plum. Report back if you manage to get something out of it : P

[real.finder]

I'm having many issues due to memory overflow.
vs feisty2 functions are so slow that run Seconds per frame not frames per second, and he made them for dvd/SD masters not even for HD

[Dogway]

Yes I know. At last I tested with a resized down to 480p clip and it kept processing 15 minutes so I closed it. Even with my optimizations and relaxed down settings (pel=2) and todays computer standards compared to 2016 (8Gb VRAM, 32Gb RAM, CUDA, etc).

Will try at another time, I need to get an output to verify it works fine, apparently F2QSharp changed some settings in the meantime.


EDIT: Did some tests with SMDegrain, motion compensating ex_median() filtering. Result are comparable at least for tr=3 and thSAD=400
# SMDegrain MC:
# STWM 9.5fps P(6) rec=false 17
# ML3Dex 20fps P(6) rec=false 31
# medianST rec=false 25fps P(6)
# MDegrain 14fps P(6) (no MC, usual method)

[real.finder]

I still need to port LimitFilter (https://github.com/HomeOfVapourSynthEvolution/mvsfunc/blob/7948c8be129bc9cb282cf24e25b3c4b77328a9e0/mvsfunc.py#L1776)and SmoothGrad (https://github.com/WolframRhodium/muvsfunc/blob/80e99100c78b96b1c925a5989259ef22c1bc6173/muvsfunc.py#L1899).

function SmoothGradPlus(clip m, int "radius", float "thr", clip "ref", float "elast", int "Y", int "UV")
{
radius = Default(radius, 16)
Y = Default(Y, 3)
Y2 = Y==3 ? 3 : 1
UV = Default(UV, 3)
UV2 = UV==3 ? 3 : 1
smooth = ex_boxblur(m, radius-1, mode="mean", Y=Y2, UV=UV2)
slimit_dif(smooth, m, ref, thr, elast, Y=Y, U=UV, V=UV)
}

convertbits(16)
SmoothGradPlus
ConvertToStacked()
DitherPost()
luma_histogram() (https://github.com/realfinder/UniversalPluginsFolders/blob/master/plugins/luma_histogram.avsi)

vs

Dither_convert_8_to_16()
SmoothGrad
DitherPost()
luma_histogram()

show that they are about 99% same (they are same but the small difference come from ex_boxblur vs Dither_box_filter16 rounding errors or something), still, SmoothGradPlus can be shorten (and it will improve performance too) by make it only one line expr (by gathering only the parts that needed for SmoothGradPlus from slimit_dif and ex_boxblur)

[Dogway]

Great, interested on the numbers ( Prefetch(cores) works best with ex_boxblur() ), I know that Dither_box_filter16() is very fast but if we are in 200+fps territory I think it's fine (EDIT: oh no rad=16!), but using ConvertBits(8) instead.

I have been researching for some time how to speed up mean blur, one method is akin to ex_gaussianblur(), by resizing down+up, but what kernel? Well with fmtconv on avs+ now we can use the "box" kernel in HBD, so that's for downscale, I'm still unsure on what to use for upscale. Maybe some resampler guru can shed some light. What I found out today is that GaussResize(p=21) is the same as BicubicResize(b=1,c=0) (a perfect cubic spline?)

[kedautinh12]

vs feisty2 functions are so slow that run Seconds per frame not frames per second, and he made them for dvd/SD masters not even for HD

feisty2 said: "i only made my scripts with placebo speed"

[Dogway]

I could approximate a mean blur with "box" resize down, mean blur rad=2, resize up in softcubic. But then I thought, is there a reasoning for using mean blur and not other blur? If you observe mean blur in high radii it's far from ideal, like image doubling. So I went and used gaussian blur with some padding.

function SmoothGradPlus(clip m, int "radius", float "thr", clip "ref", float "elast", int "Y", int "UV")
{
radius = Default(radius, 16)
Y = Default(Y, 3)
Y2 = Y==3 ? 3 : 1
UV = Default(UV, 3)
UV2 = UV==3 ? 3 : 1
pad = ceil(radius)
m = sh_Padding(m, pad, pad, pad, pad)
smooth = ex_GaussianBlur(m, (radius-1)/1.5,Y=Y2, UV=UV2)
slimit_dif(smooth, m, ref, thr, elast, Y=Y, U=UV, V=UV)
crop(pad, pad, -pad, -pad)
}


EDIT: some benchs, 126fps, with optimized slimit_dif() 183fps, without padding 285fps. Maybe we can replace GaussResize() with a border aware resizer...

[real.finder]

is there a reasoning for using mean blur and not other blur?

to make it output similar to SmoothGrad, or it will no point of use it name and it better to use SmoothGrad2 instead of SmoothGradPlus, I make it Plus since it work only in avs+ and it give almost same output

edit: btw, vs port not give similar output (again (https://forum.doom9.org/showpost.php?p=1952785&postcount=410)), so likely we will see more posts like this (https://forum.doom9.org/showpost.php?p=1952619&postcount=403) and this (https://forum.doom9.org/showpost.php?p=1952784&postcount=409) which is bad since it kinda misinformation

[kedautinh12]

I think real.finder release SmoothGradPlus cause it's similar smoothgrad and dogway will release SmoothGrad2 cause it's different smoothgrad :D

[Dogway]

Making match something for the sake of it is just an unnecessary added burden, if it does what it says and does it well, who cares? I will try to roughly match the output but I will not be nitpicky about it.
I will call it ex_deband() in any case.

I'm now working with GaussResize cropping features to add and crop borders at the same time of resizing, it's tricky, resizing down is done, up is yet.

[real.finder]

Making match something for the sake of it is just an unnecessary added burden, if it does what it says and does it well, who cares?

aside from what I said before, it necessary in case of those dll filters used in another functions that maybe very sensitive to those changes, Especial if someone replace them without know they are not same in output

[guest]

Hard to keep up to all the changes & updates...

I got SMDegrainv3.3.1d today, and got an error, seems to be OK on one PC, but no the other, both are Ryzens, but one has nVidia GPU, the other has AMD.

If I recall, it was Line 498 (489).

But everything is SO SLOW when encoding 4K with these "strong" filters :(

[Dogway]

Can you be more precise? what is your SMDegrain call?
I have tested and everything seems to work, given you updated the other scripts (in case they are being used).

SMDegrain is the fastest UHD temporal denoiser because AFAIK is the only one that implements vector scaling.
You can speed it up further using another degrain mode like ML3DEx (limited to tr=1) or STWM (tr=2) or even temporalsoften.

Anyway most of the recent changes are features rather than bugfixes.

[guest]

Can you be more precise? what is your SMDegrain call?
I have tested and everything seems to work, given you updated the other scripts (in case they are being used).

SMDegrain is the fastest UHD temporal denoiser because AFAIK is the only one that implements vector scaling.
You can speed it up further using another degrain mode like ML3DEx (limited to tr=1) or STWM (tr=2) or even temporalsoften.

Anyway most of the recent changes are features rather than bugfixes.

Well, 3.3.0d worked fine, and 3.3.1d has quite a few more lines, that's about it for changes, I made to my script.

I'd certainly like to know how to use those "modes" you mentioned above...might have to have a look tomorrow.

Cheers

[Dogway]

3.3.1d has quite a few more lines

"More lines" doesn't mean "does more", by default SMDegrain works the same as before.

[tormento]

"More lines" doesn't mean "does more", by default SMDegrain works the same as before.
Please, please, please, update your filter documentation, SMDegrain mostly.

There are some switches that are not clear to me and others rather obscure. :p

[guest]

I think I'm going to have to go back to my older filters that did a real good job, and didn't take a LONG time....these filters might be way to complex, and trying to do too much, they're just slow. :(

Reasonable quality, but at good speed !!

[Dogway]

@Pauly Dunne: Funny when almost all my changes are biased for speed, check OP for benchmarks. By the way, you just duplicated the post in the HQMC thread. You have been advised before of such demanding attitudes. We are all here to help and nothing is given for granted. So thank what is given.

@tormento: Yes I have to, it's cumbersome but with the latest additions (limitS, mode, Str) it makes more sense. Before that I want to finish ex_vibrance() and skin_qualifier() for peace of mind.

[guest]

@Pauly Dunne: Funny when almost all my changes are biased for speed, check OP for benchmarks. By the way, you just duplicated the post in the HQMC thread. You have been advised before of such demanding attitudes. We are all here to help and nothing is given for granted. So thank what is given.

You're exactly right, Dogway...

https://forum.doom9.org/showthread.php?p=1952980#post1952980

[tormento]

Before that I want to finish ex_vibrance() and skin_qualifier() for peace of mind.

SMDegrain (tr=4, thSAD=400, refinemotion=true, contrasharp=false, PreFilter=5, plane=4, chroma=true)

Script error: the named argument "sigma" to BM3D_CUDA had the wrong type (passed an array to a non-array parameter)
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.3.1d~Dogway.avsi, line 775)
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.3.1d~Dogway.avsi, line 170)

No problems with prefilter=4.

[Dogway]

Yes, you have to update (https://github.com/WolframRhodium/VapourSynth-BM3DCUDA/actions/runs/1215772307).

[tormento]

Yes, you have to update
I did not notice the update.

Thank you, it's working now.

[Dogway]

Just updated SMDegrain, LSFmod and SharpeningPack. All related to SMDegrain.
Ok, in lack of proper documentation for the time being I will brief the latest changes.

Recently there have been only three meaningful changes; denoiser mode, temporal-limiting sharpening, and ex_retinex.

ex_retinex: 'Str' is the argument that controls what kind of luminance optimization is performed. When set to 0, it only expands from TV range to PC range. When set to 1.0 or below uses ex_retinex() and over 1.0 to ex_luma_rebuild(). Default is 3.0, at this value not only a TV to PC range is performed but also some dark areas are brightened in a gentle manner. ex_retinex though uses the Multi-Scale Retinex algorithm, this is best suited for dark and/or low contrast sources and works better than ex_luma_rebuild(), since Retinex requires a range normalization the 'Str' defines the lower bound value, so when set to 0.40 it means that the retinex output will be normalized to PC range from value 40 to 255-lo/3. As always check the clip in 'show' mode to adjust values.

denoiser mode: I was developing TSMC script mod, and also having a glance to Didèe's LTSMC. When I realised, why take the whole SMDegrain to TSMC and not simply add the few lines to SMDegrain itself? So I did. Now you can use brute force denoising with all the features of SMDegrain like vector scaling, prefiltering, sharpening, etc. On the way I also had to auto activate a few switches so mode="TemporalSoften" really means "Degrain this shit out". It involved creating motion compensated frames and from there it was just a matter of including other denoisers I had created, so now you can use ex_median() temporal denoisers instead of MDegrain, some of them perform faster and with similar (better or worse) quality, just without scene change detection (should be handled by motion vectors). In today's update I included recursion mode, meaning it will denoise with past frames already denoised. It's a poor man's recursion so performance is like if called twice. Simply call like "rML3DEx" with an "r" preceding the denoiser name.

Temporal Limiting: There's not much to say about this. I reworked the temporal limiting in ex_contrasharpening(), and added it also to LSFmod, so by disabling spatial limiting with limitS=false you will enable temporal limiting. Depending on source it might work better or worse.

Some examples with the new features:

# Heavy grain denoising (mfilter to filter -blur here- where block matching was bad, aka action scenes, sudden movements). It needs a low pass recover pass that I will add in the future to avoid blending/motion blur effect.
smdegrain(tr=2,mode="temporalsoften",blksize=32,thSAD=900,contrasharp=false,refinemotion=true)
smdegrain(tr=2,mode="temporalsoften",blksize=32,thSAD=300,mfilter=ex_blur(1),contrasharp=true,refinemotion=true)

# Dealing with dark sources
smdegrain(tr=2,thSAD=300,Str=0.4,contrasharp=20,refinemotion=true)

# Using another denoiser and temporal limiting (MC frames directly fed to ex_contrasharpening() so saving a few CPU cycles)
smdegrain(tr=2,thSAD=300,mode="medianT5",contrasharp=true,limitS=false,refinemotion=true)

# Spatio Temporal denoising with recursion (STWM only uses a temporal radius of 2, so you won't see changes past tr=2)
smdegrain(tr=3,thSAD=300,mode="rSTWM",prefilter=5,contrasharp=true,refinemotion=true)

Main alternative denoisers:
Default: "MDegrain"
tr=1 -> "medianT" "MedianST", "ML3DEx"
tr=2 -> "medianT5", "STWM", "GaussT5"
tr<8 -> "TemporalSoften"

[guest]

I have a question before I take a break....

Do any of you ppl that are constantly improving & testing these filters & scripts, actually do any encoding with them ??

Like I mean a full length, 4K movie with your filters, just to see how they REALLY perform !!!

[tormento]

it only expands from TV range to PC range
Out of curiosity: what is the advantage to expand from TV to PC range if the majority of people will watch the movie on a TV set? Won't the expanded range be discarded by player/tv?

[Dogway]

@Pauly Dunne: Looks like you haven't updated LSFmod to 3.5mix, EX mod still needs to be updated to 3.5. Apparently you got bad timing because I didn't update any of my filters since 2015 and soon I will leave again. If you want to encode 4K you need to relax the settings, I'm sure you didn't live the times where filtering DVDs were at a rate of 0.5fps.

@Tormento: The range expansion is only for the motion vectors, not the denoising clip. Read from here (https://forum.doom9.org/showthread.php?p=1944794#post1944794).

[tormento]

The range expansion is only for the motion vectors
Oh, wow, thanks. Why didn't you default to 0? Perhaps not everybody wants the video to be "touched" on the dark parts.

Please, give me a hint: some movies, such as Transformers saga, has too much sharpness, giving really low compressibility ratio. FranceBB suggested me to use some vertical only MedianBlur or Blur.

What kind of command would you suggest me from your ex_plugins? And should it be applied before or after denoising?

[Dogway]

I need to see what kind of sharpness. If there isn't any overshoots a simple blur before denoising can do ( ex_boxblur(mode="weighted") with 0.5 or 0.3 value ) passed through an edge mask or even ex_smooth() but -had an idea- needs limiting. If source has overshoots (ringing or halo) you might need to use a FIR filter as once cretindesalpes pointed to me, read here (https://forum.doom9.org/showthread.php?p=1178759#post1178759)for FIR filtering.

[tormento]

I need to see what kind of sharpness
A clip is enough?

And what about str=0 vs str=3 as default?

[Dogway]

Default is Str=3 because most sources benefit from a slight bump in dark areas. Clips that might not benefit as much are generally sports, TV programs which are tuned for a bright surround viewing environment.

Yes, small clip is enough. I will try to bench ex_smooth() against it.

[tormento]

Yes, small clip is enough. I will try to bench ex_smooth() against it.

Here it is (https://send.cm/d/4ybo).

Object countours are insanely sharp, just look at sides of houses, cars and robots.

FranceBB knows the issue and solved it with some blur.

[FranceBB]

FranceBB knows the issue and solved it with some blur.

Yes I do and yes I did.
I work in post production and this generally happens with some rather sharp shots. What happens is that when I get a shot from like a Drone or even a downscaled shot from a 4K camera to FULL HD 50p and I divide it in fields to get 25i I always always always get aliasing. Even if I get 25p materials and I leave them like that and I just flag them as interlaced (so they're progressive) I still have aliasing.
For this very reason I always always always suggest to blur vertically only a bit to avoid aliasing.

blur(0, 1)

Works.

See this: https://forum.doom9.org/showthread.php?p=1875091

[tormento]

@Dogway, is it correct that setting contrasharp=true, it requires RGTools? I was almost sure that you put removegrain into ex_tools.

[tormento]

blur(0, 1)
Plain and simple :)

Let's see what Dogway will pull out of the hat. :p

[Dogway]

Yes, sorry I didn't have time to check it (battle in RGB to HSV conversion!), but I tweaked Savitzky-Golay smoothing filter a few hours ago to remove haloes. I even added clamp to ex_luts but that will be uploaded in the next few days.
function ex_smooth(clip a, int "radius", int "radiusV", bool "sharp", bool "limit", string "mode", int "UV", bool "fulls") {

rgb = isRGB(a)
isy = isy(a)
bi = BitsPerComponent(a)

rd = Default(radius, 1) # from 0 to 3
rv = Default(radiusV, rd) # from 0 to 3
sh = Default(sharp, false) # Sharp version of S-G mode
lm = Default(limit, sh) # Enable limiting for Sharp smoothing
md = Default(mode, "SG")
UV = Default(UV, rgb ? 3 : 1)
fs = Default(fulls, rgb)
rd = max(rd, 0)
rv = max(rv, 0)

rd == 3 || rv == 3 ? Assert (!sh, "S-G Sharp mode only supports radius < 3") : nop()
md == "SNN" ? Assert (rd == rv && rd < 2 && rv < 2, "SNN mode only supports radius < 2") : nop()

if (!sh) {

# Quadratic/Cubic (2nd and 3rd degree polynomials)
strv = rv == 1 ? "x[0,1] x[0,-1] + 4 * x[0,0] 5.666666666 * + x[0,-2] x[0,2] + - 0.085714286 *" : \
rv == 2 ? "x[0,1] x[0,-1] + 2 * x[0,0] 2.333333333 * x[0,-2] x[0,2] + + + x[0,-3] x[0,3] + 0.666666666 * - 0.142857143 *" : \
rv == 3 ? "x[0,3] x[0,-3] + 4.666666666 * x[0,2] x[0,-2] + 13 * x[0,1] x[0,-1] + 18 * x[0,0] 19.666666666 * + + + x[0,-4] x[0,4] + 7 * - 0.012987013 *" : ""

strh = rd == 1 ? "x[-1,0] x[1,0] + 4 * x[0,0] 5.666666666 * + x[2,0] x[-2,0] + - 0.085714286 *" : \
rd == 2 ? "x[-1,0] x[1,0] + 2 * x[0,0] 2.333333333 * x[2,0] x[-2,0] + + + x[3,0] x[-3,0] + 0.666666666 * - 0.142857143 *" : \
rd == 3 ? "x[3,0] x[-3,0] + 4.666666666 * x[2,0] x[-2,0] + 13 * x[1,0] x[-1,0] + 18 * x[0,0] 19.666666666 * + + + x[-4,0] x[4,0] + 7 * - 0.012987013 *" : ""

} else {

# Quartic/Quintic (4th and 5th degree polynomials) (sharper)
strv = rv == 1 ? "x[0,1] x[0,-1] + 15 * x[0,0] 26.2 * x[0,3] x[0,-3] + + + x[0,-2] x[0,2] + 6 * - 0.021645022 *" : \
rv == 2 ? "x[0,2] x[0,-2] + 2 * x[0,1] x[0,-1] + 9 * x[0,0] 11.933333333 * x[0,4] x[0,-4] + + + + x[0,-3] x[0,3] + 3.666666666 * - 0.034965035 *" : ""

strh = rd == 1 ? "x[1,0] x[-1,0] + 15 * x[0,0] 26.2 * x[3,0] x[-3,0] + + + x[-2,0] x[2,0] + 6 * - 0.021645022 *" : \
rd == 2 ? "x[2,0] x[-2,0] + 2 * x[1,0] x[-1,0] + 9 * x[0,0] 11.933333333 * x[4,0] x[-4,0] + + + + x[-3,0] x[3,0] + 3.666666666 * - 0.034965035 *" : ""

}

strv = md == "SNN" ? "x[0,0] X@ x[-1,1] A@ - abs X x[1,-1] H@ - abs + M@
X x[0,1] B@ - abs X x[0,-1] G@ - abs + N@ min
X x[1,1] C@ - abs X x[-1,-1] F@ - abs + O@ min
X x[-1,0] D@ - abs X x[1,0] E@ - abs + min W^
W M == A H + W N == B G + W O == C F + D E + ? ? ? 0.500001 *" : strv

rv == 0 ? a : \
isy ? Expr(a, strv ) : \
UV == 1 ? Expr(a, strv, "" ) : \
Expr(a, strv, ex_UVexpr(strv, UV, bi, rgb, fs), scale_inputs="none")
rd == 0 || md == "SNN" ? last : \
isy ? Expr(last, strh ) : \
UV == 1 ? Expr(last, strh, "" ) : \
Expr(last, strh, ex_UVexpr(strh, UV, bi, rgb, fs), scale_inputs="none")

lm && sh ? ex_clamp(last,a.ex_expand(min(1,rd),"horizontal").ex_expand(min(1,rv),"vertical"),a.ex_inpand(min(1,rd),"horizontal").ex_inpand(min(1,rv),"vertical"),0,0) : last

# ex_luts(last,a,mode="clamp",UV=uv) performance update in the next few days
}

Call like "ex_smooth(1 or 2, sharp=true, limit=true)"
If it's only vertical I guess "ex_smooth(0, 1, sharp=true)" can do.
EDIT: SantiagMod also works very good. But I see there's some ringing in the source.

EDIT2: Some alternatives:
Finedehaloer

SantiagMod(strh=2,strv=2)
a=last
ex_luts(last,last.removegrain(12,0),mode="clamp",UV=1)
ex_lutxy(last,a,"x y - abs 7 > y x y min ?")
# ex_boxblur(0.15,mode="weighted") # optional


FIR:
deconv_h = "0 -5 25 100 25 -5 0 "
mt_convolution (deconv_h, deconv_h, Y=3, U=2, V=2)

@Dogway, is it correct that setting contrasharp=true, it requires RGTools? I was almost sure that you put removegrain into ex_tools.

ExTools do have counterpart filters of RGTools, but is not SO hellish fast (90% there still), so SharpenersPack is in MIX mod. I should probably place it inside the MIX mods folder.

[anton_foy]

Sorry but what is ex_smooth? The purpose? HSV conversion sounds very interesting! Just wondering because of the way I try to make my filters.

[Dogway]

A smoothing filter, between antialiasing and blur. I just added a limiter because it was creating some 2nd order halos.

Yes, HSV conversion is proving to be more complex than I thought. There are better cylindrical models and easier to implement but I want to add HSV to TransformPack for completion and also comparison for the Skin_qualifier()

[anton_foy]

Interesting indeed although I must check into "skin_qualifier", guess that the name says it all? Please shout out if you get to finish the hsv-conversion you mentioned a while ago. I would love to try to base a denoising filter off of that colorspace.

[tormento]

Yes, HSV conversion is proving to be more complex than I thought. There are better cylindrical models and easier to implement but I want to add HSV to TransformPack for completion and also comparison for the Skin_qualifier()
So you suggest me to wait for the definitive version?

[wonkey_monkey]

Moved from the Avisynth+ discussion:

"x 0 > y x / atan A@ x 0 < y 0 >= & A pi + x 0 < y 0 < & A pi - x 0 == y 0 > pi 0.5 * y 0 < pi -0.5 * 0 0 ? ? ? ? ? ?"

I think there's some things wrong with this. If you map it to pixel coordinates (with suitable multiplication and offset) you get this:

https://i.imgur.com/O8MPTQ6.png

Note the vertical line.

Separately, the deepest conditional seems to be:

pi -0.5 * 0 0 ?

Which will always equate to zero.

While attempting to find an alternative, I've got as far as


y x / atan

dup pi - x 0 < swap2 ?

dup 2 pi * + x 0 < y 0 > & swap2 ?


though it seems to still work even when x should be 0 (I used sx instead as a pixel coordinate), which makes no sense to me.

I use "swap2 ?" for conditionals because to me it makes far, far more sense to have the condition last.

[Dogway]

Thanks for looking into it. I found the error was at the first conditional, changing it to ">=" fixes the line issue (despite originally being ">" in the wiki).

The first 0 stands for "undefined" when all the ternaries fail, and the last 0 when "x < 0" (first conditional) which seems wrong because following we have "minus than 0" comparisons, so might need to reorder the ternaries.

I implemented the compact version and also works.
ex_lutspa(mode="absolute", expr="x 255 - X@ 0 > y 255 - Y@ X / atan A@ Y 0 > pi 0.5 * X Y / atan B@ - Y 0 < 0 B - pi 0.5 * - X 0 < A pi + X 0 == Y 0 == & 0 0 ? ? ? ? ? pi 180 / * 2048 * 128 +", UV=128)


And another one without conditionals (only for the sign(x) operator):
ex_lutspa(mode="absolute", expr="x 255 - X@ 0 == 0 x dup abs / ? SX^ y 255 - Y@ 0 == 0 y dup abs / ? SY^ SX dup * Y X / atan * 1 X 0 < -1 1 ? - 0.5 * SY dup dup * - 1 + * pi * + pi 180 / * 4096 *", UV=128)
Improved (maybe) and much faster:
ex_lutspa(mode="absolute", expr="x 255 - X^ y 255 - Y^ X dup abs / SX^ Y dup abs / SY^ SX dup * Y X / atan * 1 SX - 0.5 * SY dup dup * - 1 + * pi * + 180 pi / * 128 + ", UV=128)

Please forgive my awkward normalization, I don't know what I'm doing lol

@tormento: yes, wait for the improved "HSV". I already implemented IPT model in other projects, it's miles better than HSV as it is UCS (uniform color space) in the HUE domain. But I need HSV to correlate skin HUE on HSV (33.5º) to IPT.

[tormento]

@tormento: yes, wait for the improved "HSV".
In your opinion, would I lose more details and general quality using some light vertical blur or increasing from tr=3, thsad=300 to tr=4, thsad=400?

[Dogway]

I didn't spot any staircasing so vertical blur seems very destructive to me.

[tormento]

I didn't spot any staircasing so vertical blur seems very destructive to me.
Staircasing?

[Dogway]

A type of aliasing.

EDIT: By the way:

Can't you use Taylor series? Usually with 5/6 terms you have very good approximation.

Wow, had no idea of it. At some point I searched for optimizing exponent operator and found nothing. This is a game changer. I tried though to implement to ex_contrast() without success, these days I'm a bit slow (fatigue...)
cont = 5.0
pivot = 0.5

knee = 1. / (1. + exp(cont * pivot))
shldr = 1. / (1. + exp(cont * (pivot - 1.)))
shmkn = shldr - knee
rshmkn = 1. / shmkn

# 'exp' Taylor series
exp = "X@ X 2 ^ 0.500001 * + X 3 ^ 0.166666666 * + X 4 ^ 0.041666666 * + 1 +"
Expr(last, Format("1 1 {cont} {pivot} x ymin - ymax ymin - / - * "+exp+" + / {knee} - {rshmkn} * ymax ymin - * ymin +"), "")

[tormento]

Wow, had no idea of it.
I am really happy to be of help to one of the few who have helped us all!

[tormento]

I tried though to implement to ex_contrast() without success
Look at this (https://en.wikipedia.org/wiki/Bhaskara_I%27s_sine_approximation_formula) and this (https://www.johndcook.com/blog/2021/03/21/simple-trig-approx/) too.

[pinterf]

'exp' is a valid Expr function, and is using SIMD, probably worth using it.
In your replacement code above, luckily power with small integer exponents like 1, 2, 3 and 4 are optimized internally into mul (and dup), but for larger exponent values the result is calculated using a^b = exp(b*ln(a)) which needs much more computing.

[Dogway]

I didn't want to make the snippet more complex than it is so I used "n ^", in real I'm using vars to reuse operations.
I tested with taylor series and gave a great performance improvement over 'exp' but it might not be valid for high steepness, I crafted a graph to see what was happening, I still might be doing something wrong. https://www.desmos.com/calculator/guetsfy9ww
Guess I can join another polynomial but it makes things more complex.

EDIT: just tested and yes, 'exp' is as fast. I had the notion that not when coding ex_bilateral() removing 'exp' gave a huge speed boost.

@tormento: thanks. I think cos, sin and tan are the easiest, I already adapted 'interpolation' mode in ex_blend(), the problem comes when 'x' uses derivatives and other complex functions like atan. If I'm not wrong atan(x) = tan(y) = sin(y) / cos(y)

[tormento]

If I'm not wrong atan(x) = tan(y) = sin(y) / cos(y)
Nope, the arctan function is the inverse of the tangent function: it returns the angle whose tangent is a given number.

There is a Taylor series (https://math.stackexchange.com/questions/128514/solving-the-arctan-of-an-angle-radians-by-hand) & here (https://proofwiki.org/wiki/Power_Series_Expansion_for_Real_Arctangent_Function) for it too. Look here (https://math.stackexchange.com/questions/982838/asymptotic-approximation-of-the-arctangent) also.

Explicit algorithm (https://stackoverflow.com/questions/42537957/fast-accurate-atan-arctan-approximation-algorithm) is present too.

[Dogway]

Thanks tormento, I managed to build a piecewise function for atan(x) since the Taylor series didn't converge between 0.8 and 1.65 so I built a polynomial in that section. https://www.desmos.com/calculator/bb392gsvnu
I tested on avisynth and works fine, now I will try to optimize it and benchmark, and see if I can reduce it on a case by case basis.

Here's the code and bench (400% speed increase):
Expr(last,Format(" x 255 / atan 255 *"),"") # 90


e8 = "X dup dup * X2@ X * X3@ 0.333333 * - X2 X3 * X5@ 0.200001 * + X5 X2 * X7@ 0.142857143 * - X7 X2 * 0.111111111 * + X7 X3 * 0.0909090909 * -" # up to 0.8
e16 = " X2 -0.245982 * X 1.00976 * + 0.021622 +" # up to 1.65
els = "pi 0.5 * 1 X / - 1 X3 3 * / + 1 X5 5 * / - 1 X7 7 * / - " # from 1.65 onwards
atan = "X@ 0.8 <= "+e8+" X 1.65 >= "+els+" "+e16+" ? ?"
# atan = "X@ 0.8 <= "+e8+" "+e16+" ?" # for atan([0-1])

Expr(last,Format("x range_max / "+atan+" range_max *"),"") # 413

[tormento]

Thanks tormento, I managed to build a piecewise function for atan(x) since the Taylor series didn't converge between 0.8 and 1.65
You are the most welcome. AFAIK you used a McLaurin (x=0) and not a Taylor series (where x is an arbitrary point), that's why it doesn't fit for |x| larger than 0.

[Dogway]

Yes I know, I'm currently working on cos(x) where x is pi/2 since some functions need cosines as high as pi.

Here's the Taylor series of cos(x) when x=pi/2, converges between 0 and pi.
cosTP = " pi 0.500001 * - X@ 0.00000367321 swap - X dup * X2@ 0.0000018366 * + X2 X * X3@ 0.166666666 * + X2 dup * 0.00000015305 * - X2 X3 * 0.008333333 * -"
Expr(last,Format("x range_max / pi * "+cosTP+" range_max *"),"") # 390
#Expr(last,Format("x range_max / pi * cos range_max *"),"") # 70

[wonkey_monkey]

If pi/2 < x < pi, can't you just subtract x from pi and then take the negative of the result?

Or have I misunderstood... you say you're working on cos(x) when x = pi/2, but that's just zero every time...

Possible helpful reading: http://gruntthepeon.free.fr/ssemath/sse_mathfun.h

[Dogway]

It's not cosine of pi/2 but a cosine function approximation around pi/2, so when cos(pi) it gives more accurate results than if I design the Taylor series around x=0.
Here (https://www.desmos.com/calculator/kaolz1jyav) is the desmos graph (check around x=pi )

[wonkey_monkey]

I see, so as per my previous comment: you could design your calculation around 0 < x' < pi/2, reducing x to this range appropriately first. It might be faster for the same accuracy (or more accurate for the same speed).

The purple one needs 5 powers of x, the green one only needs 3. You would basically be taking the first part of the green line (up to pi/2) and rotating it around its endpoint to extend it to pi.

https://www.desmos.com/calculator/bktakxsm7u

Edit: there is a slight discontinuity at pi/2 but you can remove that my nudging the coefficients.

[Dogway]

Yes, makes total sense, inverting the function and make it piecewise. I will bench speed and quality in case the discontinuity is visible.

[tormento]

Here's the Taylor series of cos(x) when x=pi/2, converges between 0 and pi.
Please share the desmos, it was really interesting.

[wonkey_monkey]

Using 0.0013934 (this is just a rough approximation, not a calculated value) as the x^6 coefficient should all but remove the discontinuity. The maximum error is about 0.0000924. Adding an x^8 term can make the max error almost 100x smaller.

I'll try and work on best coefficients tomorrow, if I have time.

[Dogway]

I tested and speed is 2% faster, it was already pretty fast, from 420 to 430fps. Quality wise I think it's better because it doesn't touch range extremes which are always sensible and the discontinuity is not appreciable. But if you can find a better coefficient that would be great.

There's another approximation noted by tormento, the Bhaskara I approx. but it's not as good as the six degree polynomial.
(pi^2 - 4x^2) / (pi^2 + x^2)

BTW, if I compute the Taylor series for x=pi/4 it might fix the discontinuity -> graph (https://www.desmos.com/calculator/gdug4dk5wh)
EDIT: yep, coefficient 0.001329 is almost a match, much better.


tormento: check wonkey_monkey's link (https://forum.doom9.org/showthread.php?p=1953271#post1953271) above. He includes all the three approximations, the one from my post is the purple.

[guest]

Hi Dogway (and the rest),

I asked a question back here :-

https://forum.doom9.org/showthread.php?p=1953024#post1953024

That didn't get answered.
Do any of you ppl that are constantly improving & testing these filters & scripts, actually do any encoding with them ??

Like I mean a full length, 4K movie with your filters, just to see how they REALLY perform !!!

If you're only "benching" them, that's probably NO indication on how they will work in an encoding job.

[kedautinh12]

Hi Dogway (and the rest),

I asked a question back here :-

https://forum.doom9.org/showthread.php?p=1953024#post1953024

That didn't get answered.


If you're only "benching" them, that's probably NO indication on how they will work in an encoding job.

He was answered you
https://forum.doom9.org/showthread.php?p=1953031#post1953031

[guest]

He was answered you
https://forum.doom9.org/showthread.php?p=1953031#post1953031

Hi,

I don't consider that as an answer...it's very a confusing comment, and I still can't get the latest builds to work.

And I do remember encoding DVD's at a very slow pace, my fave tool was DVD2SVCD, and I had a very powerful dual Athlon MP2600 system to churn thru it.

[wonkey_monkey]

Best coefficients I've found so far:

Up to x^6:

1 - 0.5x^2 + 0.041574811029363x^4 - 0.001292112506266x^6

Max error: 0.000019976279586 (43x better than truncated Taylor series, 4.6x better than modifying only last term to avoid discontinuity)


Up to x^8:

1 - 0.5x^2 + 0.041666666666667x^4 - 0.001387723061268x^6 + 0.000023661684925x^8

Max error: 0.000000330438621 (72x better than truncated Taylor series, 6x better than modifying only last term to avoid discontinuity)

[Dogway]

Thanks a lot. Will keep the first one for performance reasons.


EDIT: By the way, I managed to also create a Taylor (Maclaurin) series for exp(x), I know 'exp' is accelerated in Expr but it was the main cause of ex_bilateral() drop in performance ('exp' called many times) so I decided to give it a go. Well it works very well for as low as a 5th degree polynomial even in PC levels, it increased from 115fps for ex_bilateral(1) to 167fps, so it's faster than vsTBilateral(). In ex_contrast() it isn't worth it as it's only called once, and the range of action is larger (from -8 to +8 in x)

[Dogway]

Access violation, not sure if my fault or a bug in avs+:
a=FlipHorizontal()
#~ ex_blend(a,"interpolation",1,0.7)
cosTS = "X dup * X2@ 0.500001 * 1 swap - X2 dup * X4@ 0.041574811029363 * + X4 X2 * 0.001292112506266 * -" # 0.00129 to fix discontinuity at pi/2
cosT = "X@ pi 0.500001 * <= "+cosTS+" dup pi swap - -1 * ? "

Expr(last,a,"x ymin - ymax ymin - / pi * "+cosT+" 0.250001 * 0.500001 swap - y ymin - ymax ymin - / pi * "+cosT+" 0.250001 * - ymax ymin - *" ,"")

I also tried with "dup pi swap - 1 neg * ? " but the neg operator seems to not be working.


This works though:
cosTS = "X dup * X2@ 0.500001 * 1 swap - X2 dup * X4@ 0.041574811029363 * + X4 X2 * 0.001292112506266 * - dup"
cosT = "X@ pi 0.500001 * <= "+cosTS+" pi swap - -1 * ? "

Looks like I cannot dup a referenced string(?)

[wonkey_monkey]

Must be expr, I would think.

Out of curiosity, why do you use 0.500001 and 0.250001?

Very strange that just moving dup into the other string fixes it. It's all just strings that point so I don't see how expr can be to blame.

[Dogway]

Never mind, might have just tested further. "0 max" fixed it. Anyway, the string behaviour was very strange nonetheless. I might replace it with 'cos' now that is in SIMD, was just benchmarking.

About the decimals, I saw it in a vs script, so I asked here (https://forum.doom9.org/showthread.php?p=1950751#post1950751). I tested against Merge(a,b,0.5) and it happened to match Expr(a,b,"x y + 0.500001 * ")

[wonkey_monkey]

Could be down to different rounding modes or just how precision is lost in different ways with different operation orders. I don't think it's a good indication that you should use 0.500001/0.250001 in general.

[pinterf]

You cannot know if Merge is the more correct. Filters are usually using scaled-up fix point integer arithmetic which mimics floating point. e.g. 14 bit precision for intermediate calculations which fit into a 32 bit integer when dealing with 8-10 bit data.

[tormento]

I didn't spot any staircasing so vertical blur seems very destructive to me.
I have tried to reduce the bitrate of the Transformer clip that I sent with a simple Blur(0,1) and I did a couple of trials with Blur before and after SMDegrain(4,400). In both cases I had an increase in size with x264 -crf 20 -preset slow.

How could that be possible? Any idea to increase compressibility of such a sharp clip?

[Dogway]

Ok, I thought about Merge() as the reference. This (https://github.com/Asd-g/AviSynthPlus-Scripts/blob/master/IQA_downsample.avsi)is the vs script I saw it first. Anyway I will revert back all the decimal guided rounding, much easier.

@tormento: That escapes me, maybe x264 is not deterministic. Did you try my suggestions?

By the way, I managed to finish HSV conversion, took me a week! Also probably today I will upload new ExTools version with the latest changes.

[tormento]

Did you try my suggestions?
I am waiting for your latest iteration of tools :)

[wonkey_monkey]

That's possibly an attempt to avoid banker's rounding (round half-values to the nearest even number) which can introduce banding.

Compare the pixel values of the following:

expr("sx 0.5 +") # 0 2 2 4 4 6 6 8 8 10 10 ...

and

expr(sx 0.5001 + ") # 1 2 3 4 5 6 7 8 9 10 11 ...

In most cases banker's rounding is preferable as, on average, it has no bias, but in certain very specific situations it can cause banding. Dithering is one solution.

Your trig functions won't be affected because you're using so many (approximations to) irrational numbers.

[Dogway]

Wow so interesting, I see. Well when the output is dependent of such value I may use it, inside an amount of operations I don't think so because operations are done in float point. I propagated my scripts with such decimals so I need to consider it on a case by case, I guess for the Merge() behavior is a yes (?).

On another note I'm finishing IPT color space and hence Skin_Qualifier(), it's slow because it uses atan2, and I noticed some issues with my atanT() and atan2T() approximations, atan2T() has the potential to be 5 times faster (about 60fps)

[tormento]

@Dogway

I saw you released ExTools 6.0. Can I consider ex_smooth as definitive version to play with? :)

[Boulder]

I've tried testing the deblock pack, but Deblock_QED gives me an error that there is no function named Deblock_deblock. I do have the Deblock plugin installed, and I recall that calling the function like that should definitely work (removing Deblock_ fixes the issue).

[StainlessS]

Is Deblock.dll called exactly that, ie without version number or other mod ?

[Boulder]

Is Deblock.dll called exactly that, ie without version number or other mod ?

Ah, that's it. The 64-bit build is named Deblock-x64.dll so it won't work directly without either renaming or editing the function.

[StainlessS]

Not a good idea to have hyphen in dll file name, advise only "_" or alpha for 1st char, and thereafter digits too, then ".dll".

[Dogway]

I saw you released ExTools 6.0. Can I consider ex_smooth as definitive version to play with? :)

Yes sure. Also I'm the process of updating GradFun3 to ExTools, I will keep it its own script though.


By the way I'm wondering if there's a scaling option in Expr() for evaluating the expression as float instead of input bitdepth. I know the opposite is possible (evaluate as 8-bit).

[real.finder]

By the way I'm wondering if there's a scaling option in Expr() for evaluating the expression as float instead of input bitdepth. I know the opposite is possible (evaluate as 8-bit).

f32 keyword should do it

[Dogway]

Didn't work for me (https://forum.doom9.org/showthread.php?p=1953427#post1953427). I think it's not supported but pinterf has to confirm.


There are two ways currently to do this:

with ConvertBits()
ConvertBits(32)
Expr("x 0.5 - abs","",scale_inputs="none") # 187
ConvertBits(16)

And the annoying way of dividing by range_max if input is int:
Expr("x range_max / 0.5 - abs range_max *","",scale_inputs="none") # 458

[real.finder]

Expr("f32 x 0.5 scalef +","") seems work, but I see no reason for not make it work with scale_inputs, so we will wait pinterf

[tormento]

Call like ex_smooth(1 or 2, sharp=true, limit=true)
I tried to unsharpen Transformers 5, to lower encoding bitrate, but with not much success.

I remember you that this (https://send.cm/d/4ybo) was the clip.

My first attempt was

SetMemoryMax()
SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\transformers5\transformers5.dgi",ct=64,cb=64,cl=0,cr=0)
ConvertBits(16)
SMDegrain (tr=4, thSAD=400, refinemotion=true, contrasharp=false, PreFilter=4, plane=4, chroma=true)
fmtc_bitdepth (bits=8,dmode=8)
Prefetch(6)

with the x264 switches

x264.exe --crf 20 --preset slow --aq-mode 2 --colorprim bt709 --colormatrix bt709 --transfer bt709

and the result have been

encoded 5053 frames, 5.58 fps, 8195.16 kb/s, duration 0:15:05.53

As I told in my previous post, I usually get from 3000 to 4000 kbps with those settings, depending on the type of movie. So more than 8k is just a bit too much and notice that there is almost no grain or noise at all in the original video.

So I have tried to use

Blur(0,1)

just after the SMDegrain call, with same parameters, obtaining

encoded 5053 frames, 5.31 fps, 7412.87 kb/s, duration 0:15:52.00

As last attempt, I have used your new ex_smooth, just after SMDegrain call too

ex_smooth(0,1,sharp=true,limit=true)

obtaining

encoded 5053 frames, 5.16 fps, 7990.14 kb/s, duration 0:16:19.11

So, is there a better way to unsharpen and calm down that rabid movie?

[Dogway]

That was only my first suggestion, why don't you try the other (https://forum.doom9.org/showthread.php?p=1953074#post1953074) ones?
Anyway you have to consider that is an action packed snippet, maybe it's not representative of the whole movie.
For a clearer picture of the final bitrate encode chunks of it with SelectRangeEvery(), for example 2 minutes every 10 minutes or so.

[tormento]

why don't you try the other ones?
I gave the consecutive clip to the community but here I am testing with CompTest.

Where can I find SantiagMod, deconv_h and mt_convolution?

[Dogway]

This is SantiagMod, my go-to antialiaser, I simply replaced GScript with internal syntax:

###
### SantiagMod
###
### Antialiasing function by Archimedes,
### mod from original cretindesalpes's santiag.
###
###
### Requires:
###
### LoadPlugin("plugins\EEDI3\eedi3.dll")
### LoadPlugin("plugins\NNEDI2\nnedi2.dll")
### LoadPlugin("plugins\NNEDI3\nnedi3.dll")
###
### I've changed the posted function santiag a little bit.
###
### Santiag has now three parameters: type, strh and strv. Type means the antialiasing type.
### Possible values are "EEDI3", "NNEDI2" and "NNEDI3". strh is the strength for the horizontal
### anti aliasing and strv is the strength for the vertical anti aliasing. Default values are:
### type="nnedi3", strh=1 and strv=1.
###
### Contrary to the first posted function, santiag(strh=0, strv=0) now means no anti aliasing.
### I've also corected the center shift. With NNEDI3 as type, the center shift now works correct
### for all possible input clips (YV12, YUY2 and RGB24).


function SantiagMod(clip input, string "type", int "strh", int "strv") {
type = Default(type, "NNEDI3")
strh = Default(strh, 1)
strv = Default(strv, strh)

input

strh > 0 ? AntiAliasing1(type=type, strength=strh) : NOP()
TurnLeft()
strv > 0 ? AntiAliasing1(type=type, strength=strv) : NOP()
TurnRight()

function AntiAliasing1(clip input, string "type", int "strength") {
input

if (type == "EEDI3") {
EEDI3(dh=True, field=0)
for (i = 2, strength) {
EEDI3(dh=False, field=(i + 1) % 2)
}
}
else if (type == "NNEDI2") {
NNEDI2(dh=True, field=0)
for (i = 2, strength) {
NNEDI2(dh=False, field=(i + 1) % 2)
}
}
else {
NNEDI3(dh=True, field=0,qual=2)
for (i = 2, strength) {
NNEDI3(qual=2,dh=False, field=(i + 1) % 2)
}
}

Spline36Resize(input.Width(), input.Height(), 0, 0.5, input.Width(), input.Height() * 2)
}
}

mt_convolution() is from masktools2, I was too lazy to port to Expr() but if it works for you I can craft it.


By the way I have to ask if anyone is able to compile this program (https://github.com/bertdobbelaere/SorterHunter) for Windows, I need it to optimize the latest medians from 35 and up. For example "median7" has the potential to increase from current 7.8% to 12%.
median7 (48 inputs) can be composed from two 24 sorting networks and finally a batcher's odd-even merge, but before I spend more days on it I want to test if the program can find an optimized sorting network.

On another note I have been reading about Julia, a new programming (scripting?) language. As fast as C, and as "easy" as Python. I might start to learn it.


EDIT: Also I want to comment that I have been testing GradFun3 and a DFT box filter is unbeatable for large radii, so I might just keep it (if I go for back compatibility). Dither_bilateral16() can't also be replaced, I tested with vsTBilateral() at large radii but something is going on, it appears as it denoises the same as Dither_bilateral16() but when passed dithered down it doesn't deband as much, this might have to do with particular settings like flat or wmin. So rather than a revolution I might do an in-place adaptation for HBD.

[tormento]

This is SantiagMod
I have tried with the other filters.

SMDegrain (tr=4, thSAD=400, refinemotion=true, contrasharp=false, PreFilter=4, plane=4, chroma=true)
deconv_h = "0 -5 25 100 25 -5 0 "
mt_convolution (deconv_h, deconv_h, Y=3, U=2, V=2)

encoded 5053 frames, 4.98 fps, 7831.54 kb/s, duration 0:16:54.68

SMDegrain (tr=4, thSAD=400, refinemotion=true, contrasharp=false, PreFilter=4, plane=4, chroma=true)
SantiagMod(strh=2,strv=2)
a=last
ex_luts(last,last.removegrain(12,0),mode="clamp",UV=1)
ex_lutxy(last,a,"x y - abs 7 > y x y min ?")

encoded 5053 frames, 4.50 fps, 8190.77 kb/s, duration 0:18:43.92

SMDegrain (tr=4, thSAD=400, refinemotion=true, contrasharp=false, PreFilter=4, plane=4, chroma=true)
SantiagMod(strh=2,strv=2)
a=last
ex_luts(last,last.removegrain(12,0),mode="clamp",UV=1)
ex_lutxy(last,a,"x y - abs 7 > y x y min ?")
ex_boxblur(0.15,mode="weighted")

encoded 5053 frames, 4.36 fps, 7953.44 kb/s, duration 0:19:18.57

As you can see, not many changes at all.

Very hard video to compress.

[Dogway]

Put SMDegrain last, and as I commented those 5000 frames are not representative of the movie pace. Use SelectRangeEvery() to get evenly spaced chunks and get a better estimate of final bitrate. Or simply encode with target bitrate 2 passes.

[kedautinh12]

New ver santiag
https://github.com/realfinder/AVS-Stuff/blob/Community/avs%202.5%20and%20up/Santiag.avsi

[Dogway]

New ver santiag
https://github.com/realfinder/AVS-Stuff/blob/Community/avs%202.5%20and%20up/Santiag.avsi

Oh my god, who said to keep it simple. :p

[tormento]

I am having a problem with the following script

SetMemoryMax()
SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\1_48 Fantozzi\fantozzi.dgi",ct=24,cb=24,cl=0,cr=0)
ConvertBits(16)
Spresso(10,30).STpresso(10,30,4,22,8,49,1,mc=true) #last updates
fmtc_bitdepth (bits=8,dmode=8)
Prefetch(6)

with error

mt_lutxy : Error at position 3: cannot convert to number: 1.000000
(D:/Programmi/Media/AviSynth+/plugins64/STPresso-2020_0507.avsi, line 40)

Could you please convert STPresso too to use your ex_tools?

[Dogway]

I made STTWM() (https://forum.doom9.org/showthread.php?p=1950095#post1950095) 2 months ago. Did you try it?

[kedautinh12]

You don't upload that to github. I think it's only test build

[Dogway]

It was only an example script I crafted for tormento. I don't know to what degree it works or people find it useful so I can't vouch for that.


EDIT: By the way, compiled (https://github.com/Dogway/SorterHunter)SorterHunter just to know it doesn't work with inputs larger than 32, oh well...

[tormento]

I made STTWM
Sorry, it jumped out of my mind. Have I to replace into

Spresso(10,30).STpresso(10,30,4,22,8,49,1,mc=true) #last updates

or use per se?

[Dogway]

No, it's not a port it's an alternative, it uses the same concept, thresholded temporal, thresholded spatial then mix with another threshold. It's way faster. When I tested I liked the results but I normally don't use STPresso anyway. These are impulse noise denoisers (salt'n'pepper, etc) which I don't use much.

I just tested STPresso and worked for me with latest masktools and AVS+

[tormento]

No, it's not a port it's an alternative, it uses the same concept, thresholded temporal, thresholded spatial then mix with another threshold.
I am just looking for an alternative for very grainy and oddly noised video, where SMDegrain simply surrender, even with 9/900.
I just tested STPresso and worked for me with latest masktools and AVS+
Any suggested parameter? The defaults are ok?

[Dogway]

Ah for heavy grain I implemented temporalsoften into SMDegrain, it works great so far as I ported it from TSMC() mod and some bits from LTSMC() from Didée. Yes, there's some low frequency (motion) blur like, I'm working on it.

I tested STPresso with default values, maybe it might be something on your end. What line/call is giving you issues?

[tormento]

Ah for heavy grain I implemented temporalsoften into SMDegrain
Please explain me how to use it, as I can't see in the documentation.
I tested STPresso with default values, maybe it might be something on your end. What line/call is giving you issues?
I saw a new release of MaskTools is out, allowing TAB, CR and LF characters as symbol separators in expression strings.

Will give it a try.

[Dogway]

Please explain me how to use it, as I can't see in the documentation.

I explained it to you in this post (https://forum.doom9.org/showthread.php?p=1953018#post1953018) 2 weeks back.

[tormento]

I explained it to you
Oops.

I saw you mentioned ex_retinex too.

I am trying to recover some color cast, probably from film degradation, as you can see in the Retinex thread.

So, is it possible to integrate my separate line for color restoration inside SMDegrain? As you can see there, I am having some problems with too much red restoration.

[Dogway]

My version of retinex is not for color balance but for contrast enhancement.
I think there are better tools to remove the color cast like Gammac or RGBadapt. See here (https://forum.doom9.org/showthread.php?p=1632812#post1632812).

Also check here (https://youtu.be/irbZewfz-SA?t=544)from videoFred

[tormento]

My version of retinex is not for color balance but for contrast enhancement.
Thanks for your links, I will read them.

The issue with that film is that every component aged in a different manner and it's really hard to balance the hues of each one.

[guest]

What is the actual function on the "pre-filters" in SMDegrain ? eg:- Dfttest, KNLMeansCL, & BM3D.

Are they to compliment the main SMDegrain function ?, 'cause SMDegrain works pretty well, by itself !

And can the "strengths" be easily changed ?

[kedautinh12]

What is the actual function on the "pre-filters" in SMDegrain ? eg:- Dfttest, KNLMeansCL, & BM3D.

Are they to compliment the main SMDegrain function ?, 'cause SMDegrain works pretty well, by itself !

And can the "strengths" be easily changed ?

I seen default prefilter=-1 so SMDegrain works by itself
https://github.com/Dogway/Avisynth-Scripts/blob/d8634fcf489d6eaf7dee179d15dd6cb7f39240b4/SMDegrain%20v3.3.2d/SMDegrain%20v3.3.2d.avsi#L164

[guest]

I seen default prefilter=-1 so SMDegrain works by itself
https://github.com/Dogway/Avisynth-Scripts/blob/d8634fcf489d6eaf7dee179d15dd6cb7f39240b4/SMDegrain%20v3.3.2d/SMDegrain%20v3.3.2d.avsi#L164

Hi kedautinh12,

That really doesn't answer my question, though.

What is the point or the benefit of using those pre-filters ?

How to use BM3D by itself, for example.

[kedautinh12]

Hi kedautinh12,

That really doesn't answer my question, though.

What is the point or the benefit of using those pre-filters ?

How to use BM3D by itself, for example.

I seen when you prefilter some case result better in details video

You use SMDegrain(prefilter=5) to work with BM3D. You can try SMDegrain(prefilter=4) to work with KNLMeansCL, that make beter result too

[guest]

I seen when you prefilter some case result better in details video

You use SMDegrain(prefilter=5) to work with BM3D. You can try SMDegrain(prefilter=4) to work with KNLMeansCL, that make beter result too

I have found that BM3D (either CPU or CUDA) is VERY slow encoding 4K footage.

Can you change the default settings in the pre-filters ??

When I update to latest SMDegrain 3.3.2, I get errors, can only use 3.3.0.

[kedautinh12]

I have found that BM3D (either CPU or CUDA) is VERY slow encoding 4K footage.

Can you change the default settings in the pre-filters ??

When I update to latest SMDegrain 3.3.2, I get errors, can only use 3.3.0.

That cost when you use BM3D but result very well and KNLMeansCL too. You need updated all Dogway's scripts to make SMDegrain work: LSFmod, ExTools, SharpenersPack. Others like Utils-r41 and Zs_RF_Shared
https://github.com/raffriff42/AvisynthPlusUtilities/blob/master/Utils-r41.avsi
https://github.com/realfinder/AVS-Stuff/blob/Community/avs%202.5%20and%20up/Zs_RF_Shared.avsi

[guest]

That cost when you use BM3D but result very well and KNLMeansCL too. You need updated all Dogway's scripts to make SMDegrain work: LSFmod, ExTools, SharpenersPack. Others like Utils-r41 and Zs_RF_Shared
https://github.com/raffriff42/AvisynthPlusUtilities/blob/master/Utils-r41.avsi
https://github.com/realfinder/AVS-Stuff/blob/Community/avs%202.5%20and%20up/Zs_RF_Shared.avsi

It's hard to tell which one out of BM3D & KNLMeansCL does the better job..

The problem with BM3D is it is rather limited on hardware & software requirements, where as KNL is a little more versatile.

I'm pretty sure I have ALL of Dogway's latest scripts, but there are so many, and some variant's, it's hard to know which one's go with which one's.

And the other problem is there are other dependencies required, as well...depending what filters & pre-filters you choose :(

[kedautinh12]

I think it's dependencies required
https://forum.doom9.org/showthread.php?p=1953024#post1953024

[Dogway]

What is the actual function on the "pre-filters" in SMDegrain ? eg:- Dfttest, KNLMeansCL, & BM3D.

Are they to compliment the main SMDegrain function ?, 'cause SMDegrain works pretty well, by itself !

And can the "strengths" be easily changed ?
It's a bit like BM3D or KNLMeansCL ref argument. It helps to guide denoising. Think about it in terms of denoising replacement. Denoise in prefilter what you would want to denoise with MDegrain, which is more gentle.
You can define your own prefilter in the "prefilter" argument.

BM3D is not that slow, check here (https://forum.doom9.org/showthread.php?t=183066). Now if you are dealing with 4K and you want fast performance, no matter how you cut it, you need a recent PC hardware.

I will be releasing new SMDegrain version today.

[tormento]

smdegrain(tr=2,mode="temporalsoften",blksize=32,thSAD=900,contrasharp=false,refinemotion=true)
smdegrain(tr=2,mode="temporalsoften",blksize=32,thSAD=300,mfilter=ex_blur(1),contrasharp=true,refinemotion=true)
Are those supposed to run alternatively or consecutively?

[Dogway]

Consecutively. I reworked the mfilter logic for modes other than MDegrain, I will release it later today.

[guest]

It's a bit like BM3D or KNLMeansCL ref argument. It helps to guide denoising. Think about it in terms of denoising replacement. Denoise in prefilter what you would want to denoise with MDegrain, which is more gentle.
You can define your own prefilter in the "prefilter" argument.

BM3D is not that slow, check here (https://forum.doom9.org/showthread.php?t=183066). Now if you are dealing with 4K and you want fast performance, no matter how you cut it, you need a recent PC hardware.

I will be releasing new SMDegrain version today.

So when you say:- "You can define your own prefilter in the "prefilter" argument", do you mean in these lines, or somewhere else ?.

(prefilter== 4) ? inputP.ex_KNLMeansCL(a=1,d=1,h=7.0,chroma=chroma,device_type="GPU",device_id=device_id) : \ how come there's no s= option ?
(prefilter== 5) ? inputP.ex_BM3D(sigma=5,radius=1,CUDA=true,UV=Chr,fulls=fs)

I am running a 5950X with a GTX 1080, and I just find that 4K is a bit of a chore, I know it's 4x "bigger than 1080p...but :(

I also know that BM3D can be used in CPU or CUDA, BUT only on AVX2 CPU's :(.

I will get the new SMDegrain, and check all the other dependencies, update my scripts, and see if I have any problems.

Thanks

[videoh]

Consecutively. Of course. It's like deinterlacing twice because the first time wasn't good enough. :p

[guest]

It's a bit like BM3D or KNLMeansCL ref argument. It helps to guide denoising. Think about it in terms of denoising replacement. Denoise in prefilter what you would want to denoise with MDegrain, which is more gentle.
You can define your own prefilter in the "prefilter" argument.

BM3D is not that slow, check here (https://forum.doom9.org/showthread.php?t=183066). Now if you are dealing with 4K and you want fast performance, no matter how you cut it, you need a recent PC hardware.

I will be releasing new SMDegrain version today.

I compiled a reply to this post, and when I went to submit it, some "warning" flashed up about moderator clearance, and it disappeared :(

If it doesn't show up in the next day or so, I will try and remember what I wrote, and try again.

And yet, this one worked !!! :confused:

[MysteryX]

I seen when you prefilter some case result better in details video

You use SMDegrain(prefilter=5) to work with BM3D. You can try SMDegrain(prefilter=4) to work with KNLMeansCL, that make beter result too

I've done extensive testing of BM3D, KNLMeansCL and MVTools with prefilters while working on xClean (https://forum.doom9.org/showthread.php?t=183245).

What I found is that each of the 3 behave differently when it comes to prefilters. Here's a summary of the strengths and weaknesses of each, and the effect of prefilter.

MVTools
Strength: Removes a lot of noise, good at removing temporal noise.
Weakness: Can remove too much, especially with delicate textures like water.
Ref: Impacts vectors analysis but low impact on outcome

BM3D
Strength: Good at preserving fine details!
Weakness: Doesn't remove much grain. Poor temporal stability.
Ref: Moderate impact on outcome. A blurry ref will remove more grain while BM3D puts back a lot of details.
radius=1 provides nearly no benefit at huge performance cost since MVTools already does temporal analysis

KNLMeansCL
Strength: Best general-purpose denoiser
Weakness: Can blur out details and give an artificial plastic effect
Ref: Highly impacts the outcome. All prefilters benefit from running KNLMeans over it.

MVTools + BM3D
Strength: Keeps a lot of details with good denoising of fine details as well as larger grain. Single frames can look great.
Weakness: Poor noise temporal stability. The video doesn't look as good as single frames.

MVTools + KNLMeans
Strength: KNLMeans with extra denoising. Works best in most circumstances.
Weakness: Delicate textures like water or fog will suffer.

MVTools + BM3D + KNLMeans
Strength: Like MvTools+KNLMeans but preserves details with delicate textures. Works best for any kind of content tested.
Weakness: Performance and memory usage.

[guest]

It's a bit like BM3D or KNLMeansCL ref argument. It helps to guide denoising. Think about it in terms of denoising replacement. Denoise in prefilter what you would want to denoise with MDegrain, which is more gentle.
You can define your own prefilter in the "prefilter" argument.

BM3D is not that slow, check here (https://forum.doom9.org/showthread.php?t=183066). Now if you are dealing with 4K and you want fast performance, no matter how you cut it, you need a recent PC hardware.

I will be releasing new SMDegrain version today.

So despite getting all the latest updates, and updating my scripts, I still can't get them all to work happily together.

Also, have to say a big thanks for listing ALL the dependencies, for SMDegrain :)

[kedautinh12]

So despite getting all the latest updates, and updating my scripts, I still can't get them all to work happily together.

Also, have to say a big thanks for listing ALL the dependencies, for SMDegrain :)

You need say exactly your problem

[guest]

You need say exactly your problem

I need to have a bit of a cleanup...I've got so many different script builds, might need to start from scratch.

But I am a little puzzled that it all worked several builds back, but then things get shuffled around, and it produces errors.

[tormento]

MVTools + BM3D + KNLMeans
Strength: Like MvTools+KNLMeans but preserves details with delicate textures. Works best for any kind of content tested.
Weakness: Performance and memory usage.
How do you combine and apply them? Can post a sample script?

[Dogway]

I've done extensive testing of BM3D, KNLMeansCL and MVTools with prefilters while working on xClean (https://forum.doom9.org/showthread.php?t=183245).

I found BM3D to be a bit soft on edges, I think because it does some spatial filtering (3D). For video I normally avoid denoising directly in spatial domain unless it's something very specific or masked. For guiding I think it's good, in my brief comparisons motion estimation was better than KNLMeansCL, but it usually hanged when trying higher radius.

@TDS: Post the error message so I can have a look, otherwise can't help. If you want a cleanup tip, just keep core filters and move out old ones or used sparsely, then try to update them to latest version, including AviSynth.

[StainlessS]

I compiled a reply to this post, and when I went to submit it, some "warning" flashed up about moderator clearance, and it disappeared :(

If it doesn't show up in the next day or so, I will try and remember what I wrote, and try again.




TDS, you need this [in the 1st post, TextArea Cache ].:- https://forum.doom9.org/showthread.php?t=177066


First time I've had to use it and saved me about 20->30 minutes of repeated searching to re-write post after accidental deletion.

I had several different threads open on D9, and accidentally went back to previous page on the wrong tab, result: about 20 to 30 minutes of searching/writing lost.
I clicked on the TextArea Cache tool icon, and it was there just like it was before deletion, this is defo gonna be staying on all my machines, it was there just when I needed it.

It saves copy of text edit boxes of fireFox. [you can give list of websites not to cache eg your on-line banking]
Saved my bacon, perhaps do same for you one day.

[guest]

I found BM3D to be a bit soft on edges, I think because it does some spatial filtering (3D). For video I normally avoid denoising directly in spatial domain unless it's something very specific or masked. For guiding I think it's good, in my brief comparisons motion estimation was better than KNLMeansCL, but it usually hanged when trying higher radius.

@TDS: Post the error message so I can have a look, otherwise can't help. If you want a cleanup tip, just keep core filters and move out old ones or used sparsely, then try to update them to latest version, including AviSynth.

That's the plan....and I will post errors msg's too... it's just they vary from script to script.

Thanks

[guest]

TDS, you need this [in the 1st post, TextArea Cache ].:- https://forum.doom9.org/showthread.php?t=177066

Strange thing was, I had posted several replies to another user, but the one to Dogway failed :(

[guest]

@TDS: Post the error message so I can have a look, otherwise can't help. If you want a cleanup tip, just keep core filters and move out old ones or used sparsely, then try to update them to latest version, including AviSynth.

Hey Dogway, can you please reply to this post :-

https://forum.doom9.org/showpost.php?p=1954281&postcount=491

It was delayed a day or so for some reason, and it's now been allowed :confused:

And another thing, now that MysteryX has mentioned multiple pre-filters, could that be done with your scripts ??

eg:- video=SMDegrain(video,tr=6,thSAD=600,contrasharp=30,str=1.2,refinemotion=true,prefilter=4,prefilter=5)

[kedautinh12]

Oh, hi oldman, TDS is your new account? :D

[kedautinh12]

SMDegrain only support single prefilter. You need wait MysteryX port xclean to avs but you sure will complain about speed again :D

[Dogway]

OK, now I see. Yes there's no 's' argument in ex_KNLMeansCL() I added it now (also wmode) and defaulted to be the same as 'a', and it gains some sharpness back. Anyway this doesn't trigger any error whatsoever...

You can use any prefilter on your own, the 'prefilter' argument is just a commodity, it's not an end-it-all solution, for advanced prefiltering you define it yourself like:
pre=fluxsmootht(3).removegrain(11)
SMDegrain(tr=6,thSAD=500,contrasharp=40,prefilter=pre,str=1.2,refinemotion=true)

The documentation is old but it's still useful for most of the settings, so have a look at it.
Also observe the presence of 'Str' the prefilter clip is still passed the normalization stage with ex_luma_rebuild() in this case.

[guest]

Oh, hi oldman, TDS is your new account? :D

Hmmm, close, we both use RipBot264, and "we" have been helping each other trying to improve RipBot as the dev seems to have lost his interest of late.

When he mentioned that he was going to take a break, I created an account to carry on.

I'm sure he will appear every so often.

You helped him a lot, so hopefully you can continue, for me ?

But it can be a bit tedious (TDS) ;)

[kedautinh12]

Hmmm, close, we both use RipBot264, and "we" have been helping each other trying to improve RipBot as the dev seems to have lost his interest of late.

When he mentioned that he was going to take a break, I created an account to carry on.

I'm sure he will appear every so often.

You helped him a lot, so hopefully you can continue, for me ?

But it can be a bit tedious (TDS) ;)

Ok, i will help you in my possibility :D

[guest]

OK, now I see. Yes there's no 's' argument in ex_KNLMeansCL() I added it now (also wmode) and defaulted to be the same as 'a', and it gains some sharpness back. Anyway this doesn't trigger any error whatsoever...

You can use any prefilter on your own, the 'prefilter' argument is just a commodity, it's not an end-it-all solution, for advanced prefiltering you define it yourself like:
pre=fluxsmootht(3).removegrain(11)
SMDegrain(tr=6,thSAD=500,contrasharp=40,prefilter=pre,str=1.2,refinemotion=true)

The documentation is old but it's still useful for most of the settings, so have a look at it.
Also observe the presence of 'Str' the prefilter clip is still passed the normalization stage with ex_luma_rebuild() in this case.

Thanks, but I really don't understand most of this stuff. :stupid:


HOWEVER:-

#change contrasharp=true to contrasharp=(int) it errors on line 538 of SMDegrain333d

#adding lsb=true or false, wont even start Avsmeter

These didn't work for me.

#pre=fluxsmootht(3).removegrain(11)
#SMDegrain(tr=6,thSAD=500,contrasharp=30,prefilter=pre,str=1.2,refinemotion=true,lsb=true)

So, so far, so good, on the next script :)

[Dogway]

lsb settings are deprecated. They are not in SMDegrain anymore, the error message should tell you.

Also:
#change contrasharp=true to contrasharp=(int) it errors on line 538 of SMDegrain333d

Is not an error message it's the telephone game. If I had to guess you explicitly wrote "(int)" into the argument?

By the way I'm going to update SMDegrain with a fair amount of changes, nothing critical all bugfixes and improvements.
mfilter is also fixed for defined CClip inputs. Recursion now works again for medianT when tr<=2, and added a few Asserts for it. The 's' argument is added to ex_KNLMeansCL(), and passed to the prefilter which has been tuned for a bit more sharpness. ex_retinex() blacks now have been clipped instead of scaled, this respects the original intent better, also it's faster. New setting which I wanted to add since 7 years ago, 'LFR', it will restore low frequency details, specially useful for TemporalSoften, check the images below.

smdegrain(tr=2,mode="temporalsoften",blksize=32,thSAD=900,LFR=5,contrasharp=false,refinemotion=true)
smdegrain(tr=2,mode="temporalsoften",blksize=32,thSAD=300,LFR=false,mfilter=ex_boxblur(5,mode="weighted",UV=3),contrasharp=true,refinemotion=true)

source
http://i.imgur.com/DGUVkQWt.png (https://i.imgur.com/DGUVkQW.png)
before
http://i.imgur.com/fC2YWxTt.png (https://i.imgur.com/fC2YWxT.png)
after low frequency restore
http://i.imgur.com/nirP1ygt.png (https://i.imgur.com/nirP1yg.png)

[guest]

lsb settings are deprecated. They are not in SMDegrain anymore, the error message should tell you

Is not an error message it's the telephone game. If I had to guess you explicitly wrote "(int)" into the argument?

By the way I'm going to update SMDegrain with a fair amount of changes, nothing critical all bugfixes and improvements.


So sorry 'bout that, I was actually going to write "a number" but thought int would describe that.
Is not an error message it's the telephone game. If I had to guess you explicitly wrote "(int)" into the argument?

#change contrasharp=true to contrasharp=(a number) it errors on line 538 of SMDegrain333d

smdegrain(tr=2,mode="temporalsoften",blksize=32,thSAD=900,LFR=5,contrasharp=false,refinemotion=true)
smdegrain(tr=2,mode="temporalsoften",blksize=32,thSAD=300,LFR=false,mfilter=ex_boxblur(5,mode="weighted",UV=3),contrasharp=true,refinemotion=true)

I very much doubt that I will be able to use 2 SMDegrain lines like that.

[Dogway]

Why don't you post the error message or the SMDegrain call? I can't interpret what you interpret of those. Post concise information so I can give you precise answers.
SMDegrain v3.3.4d is released now. Currently LFR requires ExTools v6.1 which is unreleased because I want to solve the optimized median of 48 inputs. I might release it without it if I get stuck.

[guest]

Why don't you post the error message or the SMDegrain call? I can't interpret what you interpret of those. Post concise information so I can give you precise answers.
SMDegrain v3.3.4d is released now. Currently LFR requires ExTools v6.1 which is unreleased because I want to solve the optimized median of 48 inputs. I might release it without it if I get stuck.

OK, I thought I was explaining it as best I could, but in the future I will post the "call", then try to explain the error....not sure if I can copy the error msg.

Will get latest SMDegrain, and see how it goes.

Currently I have everything working sweet, after the "clean up" so that was worth it.

One other question:-

Can the values in red, be easily changed, or does it need to be done somewhere else ??

(prefilter== 4) ? inputP.ex_KNLMeansCL(a=2,s=2,d=1,h=7.0,wmode=0,chroma=chroma,device_type="GPU",device_id=device_id) : \
(prefilter== 5) ? inputP.ex_BM3D(sigma=5,radius=1,CUDA=true,UV=Chr,fulls=fs)

[Dogway]

You can do like this:
pre=ex_KNLMeansCL(a=2,s=2,d=1,h=7.0,wmode=0)
SMDegrain(tr=3,thSAD=300,contrasharp=30,prefilter=pre,refinemotion=true)

[guest]

You can do like this:
pre=ex_KNLMeansCL(a=2,s=2,d=1,h=7.0,wmode=0)
SMDegrain(tr=3,thSAD=300,contrasharp=30,prefilter=pre,refinemotion=true)

I'm pretty sure I cannot use a script like that...see here for an example of what my scripts need to look like (ignore the build numbers, etc)

https://forum.doom9.org/showpost.php?p=1953024&postcount=398

[Dogway]

Why not? elaborate please.

[guest]

Why not? elaborate please.

Several reasons...

The way that scripts are written for/in RipBot264.

My lack of experience & understanding of how to write / adapt scripts.

I have tried a couple of multi line scripts like that, and it just doesn't work :(

[Dogway]

Write in the RipBot264 thread on how you can adapt the above call to be read by RipBot264... I don't use any interface personally.

[Boulder]

That LFR might be useful as a generic function - do you have plans to put it in ExTools? There are often cases where a high thsad is a must but it smooths the flat, low-frequency surfaces a bit too much.

[guest]

Write in the RipBot264 thread on how you can adapt the above call to be read by RipBot264... I don't use any interface personally.

I will give that a try, but I don't expect much.

That is why I'm here.

So how do you test your scripts ?

[StainlessS]

So how do you test your scripts ?
You could try VirtualDub2, AvsPMod, PotPlayer, MPC-HC, MPC-BE, or other players [not VLC].
Timing, AvsMeter.

[Dogway]

Ah yes sorry, I use avspmod, but for encoding I simply use a custom bat file.
That LFR might be useful as a generic function - do you have plans to put it in ExTools? There are often cases where a high thsad is a must but it smooths the flat, low-frequency surfaces a bit too much.
I don't see it to fit much... if any maybe ex_repair() but it's just a lowpass comparison, for MDegrain I used an alternative (cheaper) because vectors are involved. More interesting would be to have a fast filter where you can convert to frequency domain, filter with any script and convert back. I will post the function here tomorrow (football day today : D)

[kedautinh12]

Ah yes sorry, I use avspmod, but for encoding I simply use a custom bat file.

I don't see it to fit much... if any maybe ex_repair() but it's just a lowpass comparison, for MDegrain I used an alternative (cheaper) because vectors are involved. More interesting would be to have a fast filter where you can convert to frequency domain, filter with any script and convert back. I will post the function here tomorrow (football day today : D)

Oh, i know Spain has final match in Nation League today :D

[anton_foy]

Testing the latest SMDegrain (6.0) SMDegrain(tr=3,thSAD=300,contrasharp=30,refinemotion=true) gives me this error:
Script error: Convert: Unknown chromaplacement. Line 211. Yet no problem in 5.8 version.

[guest]

Testing the latest SMDegrain (6.0) SMDegrain(tr=3,thSAD=300,contrasharp=30,refinemotion=true) gives me this error:
Script error: Convert: Unknown chromaplacement. Line 211. Yet no problem in 5.8 version.

Not sure about those version numbers...3.3.4 is the latest.

Try contrasharp=true. (or false)

[anton_foy]

Not sure about those version numbers...3.3.4 is the latest.

Try contrasharp=true. (or false)

Sorry I meant the latest version v3.3.4d but nope still the same error without contrasharp.

[Dogway]

@anton_foy: Maybe you need to update avisynth. top_left alignment was added recently.

@TDS: I think what the problem is. RipBot264 doesn't understand 'contrasharp' val type.You should use a maintained program like avspmod, etc, so at least you know that the problem might be elsewhere.

[guest]

@TDS: I think what the problem is. RipBot264 doesn't understand 'contrasharp' val type.You should use a maintained program like avspmod, etc, so at least you know that the problem might be elsewhere.

You might be right, RipBot264 is pretty "closed up" as far as it's internal coding is concerned, but I have learned recently how to add more filters & scripts.

I posted that script you sent me on the RipBot forum, and the dev (Atak) actually replied with a modified script:-

pre=ex_KNLMeansCL(video,a=2,s=2,d=1,h=7.0,wmode=0)
video=SMDegrain(video,tr=3,thSAD=300,contrasharp=30,prefilter=pre,refinemotion=true)

Contrasharp bool (True/False, default False)
Contrasharp int (0-100, default -)
Contrasharpening is a technique that compares the differences between the clip before blurring (original) and after blurring (filtered), and sharpens locally with consequent strength. By default the "sharp" clip is the input, the "after" clip is the denoised clip. Alternately, a "before" clip can be specified with CClip (See Advanced).
True: use Didée's Contrasharpening() function (ContraHD() for HD) which "Sharpens the denoised clip, but doesn't add more to any pixel than what was removed previously." In the practice you will get a slightly sharper result than the source, which is welcome.
An integer: LSFmod() will be used instead. It will be much slower, but maybe better for certain sources. Its value will serve as a contrasharpening multiplier, use one around 50 for similar strength as Contrasharpening(). LSFmod is less-suitable for HD sources.

So this could be done to ALL the pre filters, especially BM3D ??

[FranceBB]

Oh, i know Spain has final match in Nation League today :D

And... they've lost to France with a probably offside goal xD
(But yeah, I was watching the game too, so I guess Doom9 stopped for a bit while everyone was watching it xD)

[Dogway]

@TDS: Yes, you can use contrasharp to any filtering that does some blur. probably the issue you are facing is because RipBot parses all the arguments and only accepts one type per argument, in this case bool. This is something you have to discuss with them, try first with another filter that uses "val" type.

Yep, lost to Mbappe offside goal, it reminds me of Henry's hand, or PSG's financial "fair play", anyway Spanish squad is very young many playing for first time. My favourite was Italy, best national team by far in my opinion.

By the way I might upload updated ExTools later today, with a fair amount of improvements.

[guest]

@TDS: Yes, you can use contrasharp to any filtering that does some blur. probably the issue you are facing is because RipBot parses all the arguments and only accepts one type per argument, in this case bool. This is something you have to discuss with them, try first with another filter that uses "val" type.


I don't think I need to do anymore about Contrasharp, I'm happy with "true" or "false". I was actually shocked that I got a reply on the RipBot forum.

So can BM3D be setup like you did with KNLMeansCL ??

pre=ex_BM3D(video,???????)
video=SMDegrain(video,tr=3,thSAD=300,contrasharp=30,prefilter=pre,refinemotion=true)

I tried a couple of different things, but couldn't get it to work.

[Boulder]

Tried QTGMC with inputtype=2, got error "I don't know what 'fs' means.", pointing to extools.avsi at line 1191 and qtgmc.avsi at line 594.

EDIT: Fixed by adding ', bool "fs"' to ex_GaussianBlur's parameter list, but got the next error: "MergeLuma: Images must have same width and height!" pointing to lines 1189 (extools) and 882 (QTGMC).

[Dogway]

Just updated ExTools to v6.1 ans SimilarityMetrics to v1.2.

@Boulder: Try the new version as the issue was addressed a week ago but hadn't released the update. I will also have a look for the MergeLuma thing. EDIT: fixed now.
@TDS: should be working, at least works here with defaults "ex_BM3D()"

[guest]

Just updated ExTools to v6.1
@TDS: should be working, at least works here with defaults "ex_BM3D()"

I tried:- ex_BM3D(sigma=5,radius=1) from Line 195.

Didn't work :(

[Dogway]

It works for me. It requires Utils-r41 if you are processing chroma. I might remove the dependency soon. Also be sure to have the latest BM3D version.

By the way, here is the feature rich ex_makediff() when using 'aug=true', requires SimilarityMetrics which I also updated today:

https://i.imgur.com/IQ4rWmh.png

[tormento]

New setting which I wanted to add since 7 years ago, 'LFR', it will restore low frequency details, specially useful for TemporalSoften, check the images below.
Is it automatic according to tr or thsad or is there a rule of thumb to enable it?
By the way, here is the feature rich ex_makediff() when using 'aug=true', requires SimilarityMetrics which I also updated today
Could you post a sample script? It would be nice to make it accept more than one input clip to see differences with different colors. I often do compressibility checks changing tr and thsad and it would be nice how many details they steal.

[Dogway]

No, LFR is not automatic since it makes the filter slower (haven't benchamrked though). I would use it always with TemporalSoften, and also with MDegrain if you are using thSAD>500 or tr>4 but these are loose estimations. I would run tests to see the before and after (I use ex_makediff to see better) and check if it recovers enough to be worth it.

ex_makediff() already accepts two clips. Simply call like "ex_makediff(a,b,aug=true,UV=128)".
The clip output corresponds to MAE only, it's not an indication of the numbers, only to know where to look for differences. Although I haven't tested with chroma, only with luma plane, might need to rework a few things.

[guest]

It works for me. It requires Utils-r41 if you are processing chroma. I might remove the dependency soon. Also be sure to have the latest BM3D version.


OK, well, I'll have to check if I have Utils-r41 in the list, and I have trouble finding the latest builds of BM3D are...hint, hint.

Cheers

[tormento]

ex_makediff() already accepts two clips.
I was asking for more than two :)

I usually launch smdegrain with tr=3,4,5 and thsad=300,400,500 accordingly. Sometimes 6, 600 too. I'd like to look at the differences at the same time.

[guest]

It works for me. It requires Utils-r41 if you are processing chroma. I might remove the dependency soon. Also be sure to have the latest BM3D version.


I double checked, I DID have Utils-r41, but I obviously forgot something simple, as it is now working :)

However, when I went to try the CPU "version", it showed an error on line 838, of SMDegrain334d, BM3D_CPU (sigma=[s,ch?s/2:0,ch?s/2:0], radius=r, chroma=ch) which lead me to line 829, cd = Default(CUDA, false) and I changed the (CUDA,true, to false


So originally you said to change Line 195 (prefilter== 5) ? inputP.ex_BM3D(sigma=5,radius=1,CUDA=false,UV=Chr,fulls=fs)

I hope you can follow that...

[tormento]

No, LFR is not automatic since it makes the filter slower (haven't benchamrked though). I would use it always with TemporalSoften, and also with MDegrain if you are using thSAD>500 or tr>4 but these are loose estimations. I would run tests to see the before and after (I use ex_makediff to see better) and check if it recovers enough to be worth it.
I have briefly tested in some scenes and the result with fog is awful, i.e. random blocks of pure noise (read: mess :)). When my PC will finish the encoding queue, I will test more.

[Dogway]

@TDS: Nice. I tested "ex_BM3D(sigma=5,radius=1,CUDA=false,UV=3,fulls=false)" and it worked here. Do you have the latest version?

@tormento: Great! share the clip (and SMDegrain call) so I can have a look. ExTools and SMDegrain are supposed to be finished except for bugs.

[guest]

@TDS: Nice. I tested "ex_BM3D(sigma=5,radius=1,CUDA=false,UV=3,fulls=false)" and it worked here. Do you have the latest version?


Latest version of what ???

If you're referring to BM3D, I think I have "test 3"...is there newer ??

[Dogway]

test 3 is not latest, it's this (https://forum.doom9.org/showthread.php?p=1951477#post1951477).

[guest]

test 3 is not latest, it's this (https://forum.doom9.org/showthread.php?p=1951477#post1951477).

OK, thanks for that, but I think that might be the one I have...I'll check.

[kedautinh12]

No, this is last
https://github.com/WolframRhodium/VapourSynth-BM3DCUDA/actions/runs/1215772307

[Dogway]

Thanks, that's it.

[guest]

Thanks, that's it.

Were you able to get this ??

It's timestamped @ 09-09-21, older build was 04-09-21.

[kedautinh12]

My link is ver 09-09-21

[guest]

My link is ver 09-09-21

Hi, yes, but it took me a while to get to the downloadable file...

[kedautinh12]

I downloaded easy with github account

[guest]

I downloaded easy with github account

Yes, once I signed in, made all the difference :)

[tormento]

Great! share the clip
Warning! NSFW!

It's the shower scene (https://send.cm/d/5JUr) in Carrie.

It's a plain script

SetMemoryMax()
SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\carrie.dgi",ct=20,cb=20,cl=0,cr=0)
ConvertBits(16)
SMDegrain (tr=x, thSAD=x00, refinemotion=true, contrasharp=false, PreFilter=4, plane=4, chroma=true) [,lfr=true]
fmtc_bitdepth (bits=8,dmode=8)
Prefetch(6)

where x varies from 4 to 9, accordingly.

Any hint is welcome.

[Dogway]

Thanks, I didn't get block of any sorts, simply recovered low frequency grain.
You have two options, run my double temporalsoften call (double MDegrain also works), or lower the frequency restoration with LFR=8 for example.
https://i.imgur.com/ibjwVaP.png https://i.imgur.com/vHKSqlY.png


EDIT: this is a good compromise between quality and speed
smdegrain(tr=1,mode="temporalsoften",blksize=32,thSAD=900,LFR=2,contrasharp=false,refinemotion=true)
smdegrain(tr=1,mode="MDegrain",blksize=32,thSAD=300,LFR=false,contrasharp=true,refinemotion=true)

[tormento]

lower the frequency restoration with LFR=8
Does the number modify the frequency domain or the restoration intensity?

[Dogway]

The number indicates the cutoff frequency of the lowpass (it isn't parametrized but I would like to do it in the future). The higher the lower frequency is restored, the lower more high (along low) frequencies are restored. It's a low pass so that means that recovered frequencies span from 0 to cutoff frequency.

[tormento]

The number indicates the cutoff frequency of the lowpass
Do you have any idea of some meaningful numbers/cutoffs to use?

[Dogway]

Depends on source grain, the greater the grain the more low freq noise you will get. Or you can use two passes as above and get away with a lower LFR (higher cutoff).

Also I forgot about your query on ex_makediff(), can't you just do:
a=last
b=SMDegrain(tr=5, thSAD=500)
c=SMDegrain(tr=7, thSAD=700)
d=SMDegrain(tr=9, thSAD=900)
b=ex_makediff(a,b,aug=true)
c=ex_makediff(a,c,aug=true)
d=ex_makediff(a,d,aug=true)

StackHorizontal(b,c,d)

[Julek]

I remember you mentioned about reinstalling VS, so I wanted to let you know that Akarin's Expr now has sortN, which is very useful.
https://github.com/AkarinVS/vapoursynth-plugin/wiki/Expr#using-sortn-to-implement-rank-order-filters

[Dogway]

Wow that's great, too bad pinterf dismissed the sort operator. I'm still trying to figure out how to compose a 48 inputs ranking filter, I know I can use two parallel 24 inputs, then use a Batcher's odd-even sorter to merge both ranked lists, I still don't know how to set up this odd-even sorting network for 48 inputs, I need to run some tests probably in a few days. Will be the last thing to add to ExTools.

I haven't installed VS because my main urge is to finish the current scripts and its issues. Had to rebase all the scripts for the new ex_blur() gaussian to binomial fit. Then also created a MatchClip() function to remove MatchColorFormat() and its Utils-r41 dependency. Also created a few rounding functions to remove m4(), m4_sh, m(), etc and also Zs_RF_Shared.avsi dependency.

My main problem with VS is python, despite I use it for blender and a few random scripts I prefer the more direct approach of AVS+. Programming is reduced to a minimum and you only care about the algo. Python is a bit harder to read, also because the object oriented approach, a bit too verbose IMO. I will install it when I get TransformsPack to a stable release, want to play with the AI filters.

[tormento]

I haven't installed VS
Don't give in to the dark side. :p

[Dogway]

No I won't. I look forward to Julia, much faster and probably easier to read (I hate the "X if true else false" of python).

Just updated SimilarityMetrics, vsSSIM had a few issues. Also finished BSSIM.
Also updated ResizersPack, main utility is nmod(), it converts any value to a set mod size, also has an option for minimum value (ie. nmod(n, 4, 16) ). You can also use bankers' rounding. New function PadBorders() pads the clip with borders in set 'mode', as in 'Dilate', 'Mirror' or a color '$000000'. And also updated MatteCrop() to work faster, the function still needs some work.

This weekend I will have a look at MatchClip() and update all the EX/MIX mods according.

[Julek]

I have a question about your ex_ContraSharpening, why do you use the result of "ssDD range_half -" or "ssD range_half -" to add to the denoised, and the original chooses between ssDD and ssD to then add to the denoised?

ex_CS: https://github.com/Dogway/Avisynth-Scripts/blob/master/MIX%20mods/SharpenersPack.avsi#L566
CS: https://github.com/realfinder/AVS-Stuff/blob/15ec37a05e090c9438fc855b7d2df572436ccb75/avs%202.5%20and%20up/Zs_RF_Shared.avsi#L603-L606
vs-CS: https://github.com/HomeOfVapourSynthEvolution/havsfunc/blob/master/havsfunc.py#L5691-L5693

[Dogway]

The original version uses mt_adddiff() which does "ssDD range_half -" or "ssD range_half -" in itself, so their output needs to be range_half centered.
I do everything in a single step so no need to add (or keep) the centered values. This is faster.

[Julek]

But wouldn't mt_adddiff be like denoised + ssD (the whole ssD or ssDD)?

#EDIT: I guess I really didn't know how mt_adddiff works, I'll do some tests later.

[Dogway]

You want to add the pure difference. Centering the difference is to circumvent the unsigned nature of AviSynth+ int bitdepths.

mt_addiff() is actually "y range_half - x +".

[tormento]

No I won't.
Can I make a request?

Something similar to RGBAdapt but more consistent and MT friendly, plus it still needs stacked to do HBD.

I have tried AutoAdjust too but it's just too slow to be useful on HD or 4K.

You made so many useful scripts that the only two things I can see missing is a motion stabilizer (a really working one, even for micromovements of film on the spool) and levels/gamma/RGB optimizer.

Those two can help a lot with old movies, directly ported to BD and even 4K with no restoration at all.

[Dogway]

Did you try RGBAdapt or Gammac? The core principle is easy, the hard part is to work on scene bounds and temporally soften the effect to avoid chroma flicker. This has been done in plugins, I don't know of an easy way to do it at script level.

I usually apply a hue correction but at clip level so I avoid all those issues. Normally at two points, white level and grey level. Import the curve from Photoshop apply to RGB, convert to YUV and mergechroma with that.

[tormento]

Did you try RGBAdapt or Gammac?
Yes and yes. RGBAdapt is a script and I hope it could be improved, as it presents many limitations.
I usually apply a hue correction but at clip level so I avoid all those issues. Normally at two points, white level and grey level. Import the curve from Photoshop apply to RGB, convert to YUV and mergechroma with that.
Sorry, as I told, I am a noob with color spaces and conversion. Any available tutorial to the detailed steps? :)

[Boulder]

I've been getting Access Violation errors with SPresso. Extools.avsi line 3670, smdegrain.avsi line 622 and spresso.avsi line 38 are mentioned in the dialog box.

This is the script I can use to reproduce it, it usually starts occurring if I have to refresh the script often in VDub2 while tuning the settings. Closing the application and reopening it seems to help.
DGSource("columbo_s03e01.dgi")
TFM(slow=2).TDecimate(nt=1)
SPresso(bias=20, rgmodec=4, limit=0.65)
SelectEvery(400, 1)

[Dogway]

Interesting, looks like an avisynth bug. It's in the "smart" algo of ex_median(). Investigating. Read here (https://forum.doom9.org/showthread.php?p=1955161#post1955161).


EDIT: @tormento, look here (https://emalm.com/?v=l_irt)for a simple color correction method, You can do 2 points, 3, 4, etc. Do over things you know should be neutral (opening, credits...). Then export as a .cube file.

[Dogway]

@Boulder: Did it work before, in previous versions of test AviSynth+ ? Since what version does it happen? Looks like an obscure memory bug.

[Boulder]

@Boulder: Did it work before, in previous versions of test AviSynth+ ? Since what version does it happen? Looks like an obscure memory bug.

I've only started testing SPresso recently, so all tests have been done with test14. I actually forgot that there's more recent ones too, I guess I could try testing with the latest one once my current encode finishes.

[Boulder]

Looks like test20 works without issues, I tried refreshing and jumping around the video a lot but it didn't trigger anything. I'm good with that :)

[Dogway]

Ok, I was in test 20 and got the issue initially (although not today), so the bug traces back to at least test 14.

EDIT: Updated most EX/MIX mods, mainly maintenance rebasing to latest ExTools and ResizersPack.

[Boulder]

Ok, I was in test 20 and got the issue initially (although not today), so the bug traces back to at least test 14.

EDIT: Updated most EX/MIX mods, mainly maintenance rebasing to latest ExTools and ResizersPack.
I got it now with test20 as well, if it's a memory bug, no wonder it is quite random.

[Dogway]

I will update it with a workaround until it is fixed (if it gets fixed at all) since it is so critical (used in minblur) so for the time being you can edit the "smart" algo with an explicit var:

Where it says "E^ D^ H^" replace it with "E^ D^ H^ A^ A" and the bug shouldn't appear more.

[tormento]

@tormento
I was playing with lfr and contrasharpening.

The details recovered with lfr are terrific when compared to contrasharpening. How are them different and has contrasharpening any reason to exist anymore?

[Dogway]

The lowpass recovery has an issue, it recovers everything under the cutoff (not exactly but more or less), so it's as if you didn't denoise low frequencies at all, this can be problematic for very grainy sources. That's why that's another reason to run two consecutive calls, the later cleaning up the remnant low frequency noise.

Another thing is that for LFR I'm comparing the behaviour of filtering twice (the original and the lowpassed). Maybe this is not necessary at all and can do as I did with MDegrain, simply compare the lowpassed original versus the lowpassed denoised. This would greatly improve performance.

Also there might be better filters for lowpassing, bessel, butterworth, etc. There's room for improvement.

ex_contrasharpening() doesn't recover anything, it's just acutance. I like to think in terms of LFR for low freq, and contra for high freq, so they are not mutually exclusive.

[tormento]

Another thing is that for LFR I'm comparing the behaviour of filtering twice (the original and the lowpassed).
Any example to understand the behaviour is welcome. Please keep us informed!
LFR for low freq, and contra for high freq
So, wouldn't be possible to set a integer parameter for contra and call it something like hfr?

[MysteryX]

At first I get confused or turned-off when I see too many dependencies like Dogway where you need the whole bundle just to use a function.

Then I'm thinking. What you're developing is a framework kind of like jQuery or TypeScript; that allow us to write modern full-featured JavaScript on the same-old JavaScript. That gets useful when it's well-documented and standardized. It gets a pain when you need the whole framework for a few functions, and another framework for other features, and the libraries conflict with each other.

It would particularly be useful if it would provide more unified syntax/features between Avisynth and VapourSynth. Like I see you use CombinePlanes (ShufflePlanes) extensively in Avisynth.

[Dogway]

The dependency amount is something relative as I try to reduce dependencies to the max, for example yesterday I was debating whether to use nmod() in SMDegrain or not, and decided not to, because it requires ResizersPack and SMDegrain is a widely used filter, even when I think ResizersPack should be in everyone's toolbox.

For example, nmod() tries to standardize the mod functions everyone includes in their scripts, m4(), m8(), m4_sh(), modx(). The at-least-16 (x<16?16:...) that many include is wrong because the mod(n) can output a value below 16. PadBorders() tries to replace Padding(), Addborders(), PadMirror(), sh_Padding(), _Padding(), BorderMirror(), _MirrorBorders(), etc.

I'm constantly trying to reduce dependencies so I included ex_vinverse() in QTGMC, ex_retinex, ex_minblur, ex_luma_rebuild in SMDegrain, and so on, following the concept of Packs. ex_median() replaces a handful of plugins as well. I try to shift the dependencies from plugins or individual scripts to Packs, and then to 'mode' arguments. Some people have been doing this already like manyPlus, plugins_JPSDR, Utils-r4x, jdl_xxx, Zs_RF_Shared, RT_Stats, avsresize, fmtconv.

[Boulder]

Avisynth's test21 build probably broke some similarity metrics. I just ran ex_makediff and I can see the difference with aug=true, but the metric values for PlaneSSIM, PlaneGMSD and FrameMDSI are 1.0, 0.0 and 0.0 respectively. I'm not entirely sure but I think I did check those also with test20 earlier and didn't notice anything.

[pinterf]

Avisynth's test21 build probably broke some similarity metrics. I just ran ex_makediff and I can see the difference with aug=true, but the metric values for PlaneSSIM, PlaneGMSD and FrameMDSI are 1.0, 0.0 and 0.0 respectively. I'm not entirely sure but I think I did check those also with test20 earlier and didn't notice anything.
Try please with test22, there was a glitch with some property handling script functions.

[Dogway]

@Boulder: I'm updating SimilarityMetrics and ex_makediff() today, big refactor for the later. Give me some minutes.

[Dogway]

Updated ExTools and SimilarityMetrics.
Added mode arg to ex_blur, it accepts "binomial", "gaussian" and now "butterworth", this one is a mix between mean and gaussian controlled by a bias arg included as 'n', still need to default it to 6dB/oct slope.
ex_median() now allows 'thres' for all modes.
Now ex_makediff(aug=true) shows more metrics, check below (Requires Butteraugli plugin, and TransformsPack), and added a 'show' argument to decide to what clip add the metric frameprops. I didn't include VMAF because it doesn't write frameprops, I want to add other modes like 4-SSIM, MS-SSIM, DEitp, VQM, MOVIE, BRISQUE, etc.

https://i.imgur.com/xB2LVIs.png

I've been parametrizing the blur in SMDegrain to allow frequency inputs for the LFR argument. I think it's correct now but want to do a few more checks.

[tormento]

I've been parametrizing the blur in SMDegrain to allow frequency inputs for the LFR argument. I think it's correct now but want to do a few more checks.
I find lfr really great. How does it work now?

[Dogway]

I haven't updated it yet but since the frequency sample rate depends on the resolution, instead of an absolute frequency cutoff it's now relative to the resolution.

For example by setting LFR=300, you are setting the cutoff to 300Hz from the frequency sample rate, for a 1080p clip setting the value to 1920 means no lowpass/blur. Defaults account for input sample rate. It's designed so at the cutoff the attenuation is 3dB. There's another filter I designed today the "butterworth" which typically shows a better lowpass shape, default being a bandpass + lowpass with a 6dB/oct decay/slope. I still have to find how to design this decay and see if it works as good for images as it does for audio.
There's also the "brickwall" filter, it's called the "ideal filter" but it doesn't seem to work fine over images, it should correspond to a "mean" blur kernel. Haven't had the time to test each.

[Boulder]

I've now updated a number of packs, and SPresso gives me a new error message. "Expr: Not enough elements on stack to perform operation", points to extools.avsi, line 3801.

[tormento]

EDIT: ignore following... Windows 11 latest insider decided to f*ck up the Nvidia Driver. Reinstalled the latest DCH and now it is working.

I haven't updated it

Latest everything.

Script Error

KNLMeansCL: fatal error!
(clCreateContext: OCL_UTILS_UNKNOWN_ERROR)
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.3.4d~Dogway.avsi, line 892)
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.3.4d~Dogway.avsi, line 915)
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.3.4d~Dogway.avsi, line 197)

[tormento]

Another one arose:

SetMemoryMax()
SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\2_24 007 21 - Casino Royale\21casino.dgi",ct=144,cb=144,cl=0,cr=0)
CompTest24(1)
ConvertBits(16)
SMDegrain (tr=4, thSAD=400, refinemotion=true, contrasharp=false, PreFilter=4, plane=4, chroma=true, lfr=1)
fmtc_bitdepth (bits=8,dmode=8)

Script Error

Script error: There is no function named 'nmod'.
(D:/Programmi/Media/AviSynth+/plugins64/ExTools-6.2_22~Dogway.avsi, line 1330)
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.3.4d~Dogway.avsi, line 378)
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.3.4d~Dogway.avsi, line 380)
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.3.4d~Dogway.avsi, line 380)
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.3.4d~Dogway.avsi, line 385)
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.3.4d~Dogway.avsi, line 560)

It works on Ex_tools 6.1.

[Dogway]

Yes, nmod() is in ResizersPack, it replaces sh_m4() from Zs_RF_Shared. You can copy/paste it anywhere else.

I've now updated a number of packs, and SPresso gives me a new error message. "Expr: Not enough elements on stack to perform operation", points to extools.avsi, line 3801.
Shit sorry, a stupid typo. Updated.

[Boulder]

Shit sorry, a stupid typo. Updated.

Thanks, it works now :)

One more I found out, CASm with a 1080p source cropped by 210 pixels off left and 272 pixels off right, I get an error "CombinePlanes: source and target plane dimensions are different".

[Dogway]

Fixed now in ExTools, keep'em coming!! : D

[tormento]

it replaces sh_m4() from Zs_RF_Shared
So I can get rid of ZS?

[Dogway]

It's still required for sh_GetUserGlobalIMTint() but I will try to remove that dependency in SMDegrain for the next version.

EDIT: Does anybody have a build of KPassFilterCL? I need it to double check some frequency related functions. I have GCC but I can't build avs plugins with this compiler.

Visual examples for lowpass filters:

Mean (brickwall)
https://i.imgur.com/ST86ZY1l.png

Butterworth (with n: 50, 10 and 2)
https://i.imgur.com/g8HaoZHl.png
https://i.imgur.com/XShiPu1l.png
https://i.imgur.com/iMATKqfl.png

Gaussian
https://i.imgur.com/wP1CjE1l.png



Butterworth 'n' setting animated from 300 to 0:
https://i.imgur.com/huq9JF5.gif

[poisondeathray]

EDIT: Does anybody have a build of KPassFilterCL? I need it to double check some frequency related functions. I have GCC but I can't build avs plugins with this compiler.


There is a link mirror on the top right corner of this page to v0.1.0
http://avisynth.nl/index.php/KPassFilterCL

[tormento]

I have GCC but I can't build avs plugins with this compiler.
You can easily get MSVC or Intel really easily. :D

[Dogway]

I don't have space in C: and 4Gb (at the very least) for building once every half a year...

I tested the plugin but didn't work for me... It would really be great to have vcmohan F1QLimit in avisynth to preview frequency domain in 1D.

By the way I built a sharpener ex_sharp() that by chance matched unsharp(), but I added a twist by also adding 20% of the sharpness from a lower frequency cutoff as reinforcement. It's frequency based so the varY option of unsharp() is now Fc and you can use 1920 for pixel accuracy, 1920*2 for subpixel accuracy and 1920*0.5, or simply 960 to sharpen low frequency. I also finished designing the butterworth slope, first order is actually the 6dB/oct slope, very gaussian like.

[Dogway]

Just updated SMDegrain with a battery of changes. You can read the log in the commit description (https://github.com/Dogway/Avisynth-Scripts/commit/3ea0e66d1d0f43f2e05d1dab0a1b9094248b8f4e).

To expand, I repurposed the LFR argument so now it describes frequency. The higher the more higher frequencies you restore, notice the frequency is bound to clip width (typically) resolution so 300 is not the same for a 1080p clip than for a 2160p one. By default it's to set to 300Hz for 1080p, 600Hz for 1080p and so on.
I reworked the LFR code in non-MDegrain modes to use the one in MDegrain which is sensibly faster.

I included a new setting called DCTFlicker, this is a problem I described some weeks ago which is low frequency noise so I wanted to do something with it. It happened I stumbled over this (https://forum.doom9.org/showthread.php?p=1073349#post1073349) Didée's post to tackle this very same issue. For that I had to port FluxSmoothT to Expr() which was a pending task in order to expand temporal radius and include a soft threshold. FluxSmoothT is nice because it's the minimum change of a temporal average and a temporal median, along with the soft threshold it makes for a good low frequency calmer. It's not perfect, but there's no perfect solution either (reference (https://forum.doom9.org/showthread.php?p=1471841#post1471841)). That's why a multistep solution although slower might be desired. Check below with the great sample clip from tormento.

raw
http://i.imgur.com/Gr8x0a3m.png (https://i.imgur.com/Gr8x0a3.png)
DNR..........................................................................DNR+LFR
http://i.imgur.com/LPOlEfkm.png (https://i.imgur.com/LPOlEfk.png)............http://i.imgur.com/GEsqelEm.png (https://i.imgur.com/GEsqelE.png)
DNR+LFR+DCTR........................................................DNR+LFR+DCTR (tailored: multi-step)
http://i.imgur.com/t65diLpm.png (https://i.imgur.com/t65diLp.png)............http://i.imgur.com/kY4rARZm.png (https://i.imgur.com/kY4rARZ.png)

Call for the multi-step solution
pre=smdegrain(tr=1,mode="temporalsoften",blksize=32,thSAD=900,LFR=400,prefilter=5,DCTFlicker=true,contrasharp=false,refinemotion=true)
smdegrain(tr=2,mode="MDegrain",blksize=32,prefilter=pre,thSAD=400,LFR=false,contrasharp=true,refinemotion=true)

[kedautinh12]

I seen DNR is best result just look detail of bath towel in left-top corner

[Blankmedia]

I have some problem with oyster_deringing

I've added float "h" in the function :

oyster_Deringing(clip src, clip ref, int "radius", float "h", float "sigma", float "mse0",


I've change the lesser or equel for greater or equal to 0 here:
Assert(h>=0, "Oyster.Deringing: h has to be greater than 0!")

Now "strength2" isn't being defined before being called in line 364.

If I put strength2 = 1.1 before the function call in line 364 I get script error:invalid arguments to function oy_loop.

[Dogway]

Thanks for the report! I will have a look and report back, but Oyster and Plum are both WIP (work in progress) so I'm not positive it works even when fixed. To start with the BM3D plugin isn't fully ported so many settings are missing which may be critical for the correct output, and second I'm not sure mvtools is as robust in AviSynth+ as it is in VapourSynth, one example is 32-bit support which has some issues as reported here (https://forum.doom9.org/showthread.php?p=1956391#post1956391).


EDIT: ok updated (https://github.com/Dogway/Avisynth-Scripts/blob/master/EX%20mods/DeblockPack.avsi). Good luck making it work, it hangs on my rig. I guess iterating KNLMeansCL four times isn't actually avisynth friendly.

[tormento]

Check below with the great sample clip from tormento.
A great sample from a great movie.

Anyway, look at the lower white part corner of the top left towel.

Detalis disappear everywhere but in the LFR only.

[Dogway]

Sorry guys, but I don't see what you guys are seeing.

DNR..........................................................................DNR+LFR+DCTR
http://i.imgur.com/LPOlEfkm.png (https://i.imgur.com/LPOlEfk.png)............http://i.imgur.com/t65diLpm.png (https://i.imgur.com/t65diLp.png)

Anyway I have some ideas to improve the DCTFlicker pass.

On another note, yes MatchClip() which is required for prefilter=5 and some other functions as a replacement for Utils-r41's MatchColorFormat() is not yet released (due in TransformsPack). I'm trying to catch up with all the frameprops changes by pinterf.

[tormento]

Sorry guys, but I don't see what you guys are seeing.
https://i2.lensdump.com/i/g4oTZq.md.png (https://lensdump.com/i/g4oTZq)

This part (present in raw) plain and simply disappears. The only one with some hint of it is DNR+LFR only.

Plus, I am having the error:

Script error: There is no function named 'MatchClip'.
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.3.5d~Dogway.avsi, line 887)
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.3.5d~Dogway.avsi, line 217)

Perhaps I have deleted some necessary plugin. Which is which? :)

[kedautinh12]

I seen only DNR better than DNR+LFR in perfome part of bath towel deatails

[Blankmedia]

My pleasure! I like seafood so it was compelling to use.

[Dogway]

Ok, maybe I did wrong by presenting all the steps, but I like to keep transparent always. My point was DNR+LFR+DCTR improving over DNR/DNR+LFR, which I think it does. There are dozens of different ways to fix the pointed issue, one of them is raising the cutoff but it wasn't my intention to show an example of perfect denoising, because it simply doesn't exist as stated by Didée. I'm trying hard and have some things in mind to improve low frequency denoising.

@tormento, read my previous post. MatchClip is included in yet-to-be-released TransformsPack, which got delayed due to latest avisynth changes in frameprops. Hopefully I can release tomorrow. It replaces MatchColorFormat from Utils-r41 so one less dependency. It's only needed for prefilter=5 though.

[tormento]

read my previous post
Yep I read.

Please, next time, don't implement functions you need to write yet :)

[Dogway]

It's written but not released. I was holding back all my scripts for over a week solely because MatchClip (read about the conventions disaster here (https://forum.doom9.org/showthread.php?p=1956654#post1956654)), so I pulled the trigger.

If you are too desperate use this (disclaimer, use latest avs+):

# Matches 'a' to 'b' clip format (size, length, color model, Jab, bitdepth)
function MatchClip (clip a, clip b, string "kernel", string "matrix", bool "length", bool "fast") {

krn = Default (kernel, "Spline16")
fst = Default (fast, true)
mfc = Default (length, false) # Match also framecount

YUY2 = b.IsYUY2()
rgbb = b.isRGB() rgba = a.isRGB()
w = b.width() wa = a.width()
h = b.height() ha = a.height()
fc = b.FrameCount() fca = a.FrameCount()
p_type = b.PixelType() p_typea = a.PixelType()
bib = b.BitsPerComponent() bia = a.BitsPerComponent()

isHD = (w > 1099 || h > 599) isHDa = (wa > 1099 || ha > 599)
isUHD = (w > 2599 || h > 1499) isUHDa = (wa > 2599 || ha > 1499)
mat = Default(matrix, rgbb != rgba ? isUHD ? "Rec2020" : isHD ? "Rec709" : "Rec601" : Undefined())

a
bib > bia ? ConvertBits(bib, fulls=rgba) : last

if (!fst) {

ConvertFormat(w, h, fmt_i=p_typea, fmt_o=p_type, cs_in=mat, cs_out="", kernel=krn, gamma=false, tv_range_in=!rgba, tv_range_out=!rgbb, fulls=rgba, fulld=rgbb, show=false)

} else {

rgbp = rgbb && b.isPlanar()
isScl = w != wa || h != ha
mod = rgba && rgbb || isy(a) && isy(b) ? 1 : 2

yv411 = b.isYV411() yv411a = a.isYV411()
yv444 = b.is444() yv444a = a.is444()
fmt = Format_fuzzy_search(b, p_type, bib)
fmta = Format_fuzzy_search(a, p_typea, bia)
bc = bicubic_coeffs(krn)
krn = bc[1]>=0. ? "Bicubic" : krn

cplace = rgbb || yv444 || yv411 ? "MPEG1" : isUHD ? "top_left" : "MPEG2"
cplacea = rgba || yv444a || yv411a ? "MPEG1" : isUHDa ? "top_left" : "MPEG2"
not42 = LeftStr(fmta[1],2) != "42"

# Match dimensions
isScl ? w > h ? RatioResize(w,mode="adjust2w",kernel=krn,mod=mod,b=bc[0],c=bc[1]) : \
RatioResize(h,mode="adjust2h",kernel=krn,mod=mod,b=bc[0],c=bc[1]) : last
PadResize(w,h,mode="dilate", mod=mod)

# ConvertBackToYUY2 bug: https://forum.doom9.org/showthread.php?p=1038027#post1038027
isy(b) ? rgba ? DotClip([0.298967,0.586421,0.114612]) : ConvertToY(mat) : \
rgbb ? rgbp ? Eval( "ConvertToPlanarRGB (mat,interlaced=false"+(yv411a ? "" : ",ChromaInPlacement=cplacea")+", chromaresample=krn)") : \
Eval( "ConvertTo" +string(fmt[3])+"(mat,interlaced=false"+(yv411a ? ")" : ",ChromaInPlacement=cplacea, chromaresample=krn)")) : \
yv411 ? Eval( "ConvertToYV411 (false,mat," +(not42 ? "" : """ChromaInPlacement=cplacea,""")+"chromaresample=krn)") : \
YUY2 && rgba ? Eval("""ConvertBackToYUY2 (mat """) : \
Eval( "ConvertToYUV"+string(fmt[1])+"(false,mat," +(not42 ? "" : "ChromaInPlacement=cplacea,")+" chromaresample=krn"+(yv444 ? ")" : ",ChromaOutPlacement=cplace)"))
}

bia > bib ? ConvertBits(bib, dither= 1, fulls=rgbb) : last
mfc ? fca > fc ? Trim(last, fc-1) : last+BlankClip(b,fc-1) : last

# Planar to Interleaved
YUY2 && !rgba ? ConvertToYUY2() : last }

[tormento]

If you are too desperate use this
Not desperate but I can't simply use prefilter=5 with newer features of SMDegrain.

It works, thanks.

[tormento]

use this
Another needed function:

Script error: There is no function named 'Format_fuzzy_search'.
(D:/Programmi/Media/AviSynth+/plugins64/MatchClip~Dogway.avsi, line 35)
(D:/Programmi/Media/AviSynth+/plugins64/MatchClip~Dogway.avsi, line 56)

[Dogway]

That's in TransformsPack. I'm finishing the script (QC) just waiting for pinterf input so I don't do things twice.

[Dogway]

Updated DCTFilter, hope it's fine now:

http://i.imgur.com/kTUnI6cm.png (https://i.imgur.com/kTUnI6c.png)

I will upload tomorrow with TransformsPack, which isn't finished of course but it's fine to play with already.

[tormento]

Updated DCTFilter, hope it's fine now
Much better. Thanks.

Did you alredy implemented atan2 expr from Avisynth+ 3.7.1 test build 27 (20211117) in your scripts? I saw some legacy comments but the functions are still there.

[Dogway]

Yes, I forgot to add in the changelog. So far it's only used for the cylindrical models in TransformsPack. It's a tiny bit faster than my approx. but it's internal and more precise so a winner for me. I don't think 'atan' is yet optimized but I didn't see it used in any of my scripts.

I will keep the legacy functions for reference, this is also some kind of algo spread sheet for learning.

[Dogway]

By the way, I want to introduce SoftLimiter() a filter I have been working on for the last days. It's an inclusive limiter, that means not only it drops out out of range values, but also will include out of range values in a rolloff manner. This is adaptive in the sense that further out of range values will rolloff stronger.
There's also a dynamic mode where you don't need to input the range floor/ceil after inspection. It uses ShowChannels() which works only over 8-bit so precision is not very granular.
dyn=0 is static (default), dyn=1 uses accumulative loose min/max, which is a set and forget safe mode, dyn=2 is more granular and probably better when the source range floor/ceil varies, and dyn=3 which is very granular.

https://i.imgur.com/ffXj0dj.gif

For comparison this is SoftLimiter2() by raffriff42

https://i.imgur.com/9cjbzSF.gif

And here an example with a tailored sample with range floor at 11 and ceil at 245:
Limiter()
http://i.imgur.com/4aZcTGpm.png (https://i.imgur.com/4aZcTGp.png)
SoftLimiter()
http://i.imgur.com/N8ifh7Jm.png (https://i.imgur.com/N8ifh7J.png)


I also optimized the HSV, IPT and OkLab conversion functions and they are ready to shine, now work over any bitdepth.
Here's a comparison for the hue perceptual uniformity. Since Luma and Chroma channels are mapped differently I matched those by eye.

HSV
https://i.imgur.com/wurpiNS.png
IPT
https://i.imgur.com/JdAUcxA.png
OkLab
https://i.imgur.com/LpIuxYN.png

[tormento]

I also optimized the HSV, IPT and OkLab conversion functions and they are ready to shine, now work over any bitdepth.
:thanks:

[VoodooFX]

And here an example with a tailored sample with range floor at 11 and ceil at 245...
Looks nice. What is that Limiter() there? Could you compare SafeColorLimiter (https://github.com/FranceBB/SafeColorLimiter) too?

[Dogway]

Limiter() is there just for comparison purposes, it simply chops off out of range values. SafeColorLimiter() is the same as Limiter() but with HBD support so I didn't add.
There are still a few small bits that can be improved, like 'tolerance' response, pass arguments in dynamic mode, or finer precision in the min/max values, so I will refine those at a later point.

[StainlessS]

What is that Limiter() there?
Are you asking what the Limiter() plugin is, if so then here[its avs builtin]:- http://avisynth.nl/index.php/Limiter

[VoodooFX]

[its avs builtin]:- http://avisynth.nl/index.php/Limiter
Now I remember using it for something few years ago...
What ColorYUV(levels="pc->tv) does in comparison?

[poisondeathray]

Now I remember using it for something few years ago...
What ColorYUV(levels="pc->tv) does in comparison?


ColorYUV(levels="pc->tv") does the same as levels(0,1,255,16,235,false) - it scales the clip 0-255 to 16-235. Notice slope of curve is changed, and "banding" is introduced as some values are duplicated (steps in the waveform)

Limiter just cuts off values, all values <16 become 16. >235 become 235. The slope of curve in values 16-235 is unchanged

https://i.postimg.cc/CL8QnRWC/coloryuv-levels-limiter.png

soft rolloff is a welcome adjustment option, instead of "harsh" transitions

[Dogway]

Are you asking what the Limiter() plugin is, if so then here[its avs builtin]:- http://avisynth.nl/index.php/Limiter

Ah so nice, I included a tweaked panel of the debug example in the wiki.
Also updated SoftLimiter() with a straight input remap. The curve fitting wasn't accurate plus I assessed it in 8-bit.

http://i.imgur.com/Unv9YULm.png (https://i.imgur.com/Unv9YUL.png)

[Boulder]

Testing SoftLimiter(show=true) gives me the error "StackVertical: image widths don't match", lines 2235 and 2239.

[Dogway]

Thanks, what I presumed, incompatibilities with mod sizes. Will have a look

EDIT: Fixed. Test now, I also optimized the code a bit.

[Shinkiro]

SMDgraim broken
https://i.imgur.com/QoNcvIZ.jpeg
I can't figure out what he wants from me.

[Dogway]

Strange, what version of avs+ are you on? propNumElements() works for me outside runtime environments.

[Shinkiro]

Dogway
3.7.1 r4351

SMDegrain v3.3.4d works without errors, other versions have not checked

[Dogway]

You mean r3351? Can you try newer version? I think props for frame index#0 was added recently.
Try test build 27 from its thread (https://forum.doom9.org/showthread.php?t=181351).

[Shinkiro]

With the new version it works, thanks!

[tormento]

I was trying to clean up the BD of Brazil (small clip here (https://send.cm/d/6JAx)) from excessive noise and I am really torn between

SMDegrain (tr=6, thSAD=600, refinemotion=true, contrasharp=false, PreFilter=4, plane=4, chroma=true, LFR=true, DCTFlicker=true)

and

SMDegrain (tr=6, thSAD=600, refinemotion=true, contrasharp=false, PreFilter=5, plane=4, chroma=true)

The first gives more detail behind fog or clouds but it has some dancing noise or textures in the most disturbed areas of the video, mostly visible in clouds or walls.

The second is more stable but loses some details.

The bitrate of both is similar, i.e. 3716 kb/s vs 3623 kb/s (on whole movie).

Some years ago a plugin to calm grain (I can't remember the exact name) was out and I was used to implement it in my scripts.

Can you stabilize in some way more the noise before reapplying with LFR=true, DCTFlicker=true?

Moreover I have some slight color change between the two, such as on the falling pictures on this clip (https://send.cm/d/6JCG):

https://i3.lensdump.com/i/gSAVYH.md.png (https://lensdump.com/i/gSAVYH)https://i2.lensdump.com/i/gSAQqx.md.png (https://lensdump.com/i/gSAQqx)

Look at the pictures falling on the left.

[Dogway]

I will have a closer look later today or tomorrow with more time since currently I'm porting NIS sharpen but I made some templates for noisy sources, maybe you can try, tweaking the parameters in red.

Mild to strong:
pre=smdegrain(tr=1,mode="temporalsoften",blksize=32,thSAD=900,LFR=400,prefilter=5,DCTFlicker=true,contrasharp=false,refinemotion=true)
smdegrain(tr=2,mode="MDegrain",blksize=32,prefilter=pre,thSAD=400,LFR=false,contrasharp=true,refinemotion=true)

Very strong (ie. 300, 21 grams):
smdegrain(tr=2,mode="temporalsoften",blksize=32,thSAD=900,LFR=300,DCTFlicker=true,contrasharp=false,refinemotion=true)
smdegrain(tr=1,mode="MDegrain",blksize=32,thSAD=300,LFR=false,contrasharp=true,refinemotion=true)

For calming grain I think the most popular were FluxSmoothT and STPresso. There were Deen, etc but they are older and based on the same concept (thresholded medians). You can try those or some of my implementations like ex_FluxSmoothT() or STTWM(), or ex_FluxSmoothST() if using as prefilter.

[Boulder]

Thanks, what I presumed, incompatibilities with mod sizes. Will have a look

EDIT: Fixed. Test now, I also optimized the code a bit.

I'm still getting the error with RC27. Tried without any cropping (i.e. 1080p) and with 240 pixels cropped off the left and right sides.

[Boulder]

Ex_makediff also seems to be broken, "I don't know what 'fmt' means." is the error, lines 1895 and 1910 in transformspack.avsi.

Tested by loading the 1080p source and with these as the only lines after that:
ConvertBits(16)
ex_makediff(last, SPresso(bias=20, biasc=40, rgmodec=4, limit=0.3, limitc=0.6), aug=true)

There's something strange happening with the SPresso limit, if I use limit=0.29, the output seems to change drastically.

[Dogway]

Thanks, yes stupid typo for 'aug=true'. Should be format=PixelType(RGB) in TransformsPack

As for SoftLimiter() debug panel I tested with decimal resizing (using RatioResize) for a few minutes and I had no issues. What's the exact resolution of the clip so I can check?

I did some tests for SPresso and didn't get such behaviour, the parameter response was smooth so it might be clip dependent. I would need a small sample.

[Boulder]

As for SoftLimiter() debug panel I tested with decimal resizing (using RatioResize) for a few minutes and I had no issues. What's the exact resolution of the clip so I can check?
It's 1920x1080, reproducable if I just load the source with DGSource and then use SoftLimiter(show=true).


I did some tests for SPresso and didn't get such behaviour, the parameter response was smooth so it might be clip dependent. I would need a small sample.

Try with this one. The same one can also be used for the SoftLimiter test.
https://drive.google.com/file/d/1oPbCKTTqRbNP2rLejg-WdvIIZvdxgj4g/view?usp=sharing

You can use this script to see how it looks, if you change the value of limit between 0.29 and 0.3, the difference is quite big compared to 0.3 and 0.31. It doesn't seem to matter if the source is 8 or 16 bits.

DGSource("spresso_testclip.dgi")
limit = 0.3
convertbits(16)
ex_makediff(last, mcdegrainsharp(last, c2=SPresso(bias=20, biasc=40, rgmodec=4, limit=limit, limitc=limit*2), frames=1), aug=50, uv=128)

Function MCDegrainSharp(clip c, clip "c2", int "frames", float "bblur", float "csharp", bool "bsrch",bool "Precise",Int "Limit",Int "LimitC") {
# From:- http://forum.doom9.org/showthread.php?p=1737045#post1737045
# Based on MCDegrain By Didee, http://forum.doom9.org/showthread.php?t=161594
# Also based on DiDee observations in this thread: http://forum.doom9.org/showthread.php?t=161580
# "Denoise with MDegrainX, do slight sharpening where motionmatch is good, do slight blurring where motionmatch is bad"
# In areas where MAnalyse cannot find good matches, the blur() will be dominant.
# In areas where good matches are found, the sharpen()'ed pixels will overweight the blur()'ed pixels
# when the pixel averaging is performed.
#
# Mod by StainlessS to add Precise, 3 Sept 2015.
# Mod by StainlessS, Default bblur mod for HD. bs (blocksize) mod for HD. Added Limit, LimitC. 17 Feb 2020.
frames = Default(frames, 2)
# bblur = Default(bblur, (c.width>1920) ? 0.75 : (c.width>1280) ? 0.7 : (c.width>960 ) ? 0.65 : 0.6)
csharp = Default(csharp, 0.6)
bsrch = Default(bsrch, true)
Precise= Default(Precise,False) # Use MRecalculate
Limit = Default(Limit,255)
LimitC = Default(LimitC,Limit)

bs = (c.width>1920) ? 32 : (c.width>1280) ? 24 : (c.width>960 ) ? 16 : 8
super = bsrch ? c2.MSuper(pel=2, sharp=1) : c.MSuper(pel=2, sharp=1)
super_rend = c.sharpen(csharp).MSuper(pel=2, sharp=1,levels=1) # Only 1 Level required for sharpened Super (not MAnalyse-ing)
backward_vec1 = MAnalyse(super, isb = true, delta = 1, blksize=bs, overlap=bs/2)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, blksize=bs, overlap=bs/2)
backward_vec1 = (Precise) ? MRecalculate(super, backward_vec1, blksize=bs/2, overlap=bs/4,thSAD=100) : backward_vec1
forward_vec1 = (Precise) ? MRecalculate(super, forward_vec1 , blksize=bs/2, overlap=bs/4,thSAD=100) : forward_vec1
#
(frames<=0) ? c :\
(frames==1) ? c2.MDegrain1(super_rend, backward_vec1,forward_vec1,thSAD=400,Limit=Limit,LimitC=LimitC) :\
(frames==2) ? c2.MDegrain2(super_rend, backward_vec1,forward_vec1,backward_vec2,forward_vec2,thSAD=400,Limit=Limit,LimitC=LimitC) \
: c2.MDegrain3(super_rend, backward_vec1,forward_vec1,backward_vec2,forward_vec2,backward_vec3,forward_vec3,thSAD=400,Limit=Limit,LimitC=LimitC)
return(last)
}

[Dogway]

Ok uploaded an update. The debug panel in SoftLimiter() is hopefully fixed. Also fixed the typo for ex_makediff(aug=true).
SharpenersPack is also updated with a new sharpener NVSharpen(), this is Nvidia's NIS sharpener, but I didn't like it too much. And reworked/optimized SSSharpenEX() for good I hope.

Now I will have a look at the SPresso thing and Tormento's clip.
I have a somewhat big change pending in the case masks should use fullscale bitdepth, just waiting for confirmation. In my opinion it shouldn't despite masks being PC range (except when source is also PC range).

[tormento]

SharpenersPack is also updated with a new sharpener
Did you work out some other unsharpener? I have tried many solutions but I don't see lot of improvements on bitrate output side. Always too high.

P.S: Do you plan to introduce the new development MVTools options soon or will you wait for a stable release?

[Dogway]

I ported new nvidia's sharpener, read its blog (https://www.nvidia.com/en-us/geforce/news/nvidia-image-scaler-dlss-rtx-november-2021-updates/), here the repo (https://github.com/NVIDIAGameWorks/NVIDIAImageScaling/tree/main/NIS). It's open source to compete against AMD's FSR. Someone else can port their scaler as that isn't possible with avisynth syntax.

I didn't like it much though, it's a bit naive, simple minmax edge mask that is later discretized, minmax limiting and a simple unsharp mask fixed at 3x3 window.
Bitrate increase is understandable, you are increasing acutance which raises SAD, but when I encode I only care about output quality. I guess if you want to maximize quality for bitrate using LSFmod's options is best suited because you can tell it to localize sharpening only where it cares most.

I didn't read much of the new MVTools developments by DTL, I read about some concessions on using only 8x8 blocks, that's not optimal for HD, furthermore I would really like to see adaptive block sizes like H.265, a clear sky wouldn't need less than 32 block sizes whereas a city landscape would do better with 8. I will wait for stable.

I will have a look now to the pending issues.

[tormento]

I ported new nvidia's sharpener
I wrote _un_sharpeners :)

Can't find any suitable to some excessively sharpened movies.

[Dogway]

ah, that's easy, use blur with clamping. ex_luts("clamp") as alternative to dehaloers or custom convolutions.




You can use this script to see how it looks, if you change the value of limit between 0.29 and 0.3, the difference is quite big compared to 0.3 and 0.31. It doesn't seem to matter if the source is 8 or 16 bits.

I'm testing in frame 46 with:
limit = 0.30
convertbits(16)
a=SPresso(bias=20, biasc=40, rgmodec=4, limit=limit, limitc=limit*2)
ex_makediff(a, aug=100, UV=128)
And there's a barely noticeable change between 0.29 and 0.30, compared to 0.30 to 0.31. But I mean this is augmented x100.
When using your call, yes some blocks seem to change more with 0.29 to 0.30 but this also happens with with older SPresso scripts, this is the nature of hard thresholding/limiting. I implemented soft thresholding on a few of my filters (not all). Anyway, I'm going to use old=true to my ex_MinBlur() call so it's an exact match to older SPresso. I will edit the script again when pinterf adds the sign operator.

[Boulder]

I'm testing in frame 46 with:
limit = 0.30
convertbits(16)
a=SPresso(bias=20, biasc=40, rgmodec=4, limit=limit, limitc=limit*2)
ex_makediff(a, aug=100, UV=128)
And there's a barely noticeable change between 0.29 and 0.30, compared to 0.30 to 0.31. But I mean this is augmented x100.
When using your call, yes some blocks seem to change more with 0.29 to 0.30 but this also happens with with older SPresso scripts, this is the nature of hard thresholding/limiting. I implemented soft thresholding on a few of my filters (not all). Anyway, I'm going to use old=true to my ex_MinBlur() call so it's an exact match to older SPresso. I will edit the script again when pinterf adds the sign operator.

Thank you for the explanation. In fact, it's fine to keep it as it is now that I know how it is. I often use quite small values for limit in my denoiser to just push the hardest grain edges back a bit.

[tormento]

use blur with clamping. ex_luts("clamp")
Is that the exact command to use? ex_luts("clamp")?

[Boulder]

I found a rather peculiar problem with SoftLimiter and apparently multithreading.

This script causes strange seeking which quite often crashes RequestLinear with "RequestLinear: internal error (frame not cached)!". I'm currently on test 27 of Avisynth+'s latest builds.
DGSource("source.dgi")
RequestLinear(clim=100, debug=true)
SoftLimiter(dyn=2)
Prefetch(threads=24, frames=12) # good for 12-core 3900X

https://pastebin.com/TgjREc0f, see for example line 588.

[videoh]

Since DGSource() supports random access, why is RequestLinear() needed? What is it doing?

[Boulder]

This strange glitch, https://github.com/pinterf/mvtools/issues/37, does not occur if I use RequestLinear there.

[videoh]

DGIndex and DGIndexNV both use the same random access logic, so if you can find a way to produce this error with DGIndex then perhaps pinterf will be able to look at it. We should chip in to get him an nVidia GPU, if anyone can find one.

[Dogway]

Is that the exact command to use? ex_luts("clamp")?

It's the same that I posted (https://forum.doom9.org/showthread.php?p=1953074#post1953074) a while ago, antialiasing, dehaloers, masked blurs are all good for that.

[tormento]

It's the same that I posted a while ago.
Unfortunately none of them gave good results on movies such as Transformers, Pacific rim, etc... where the CGI sharpness was exasperated to increase "realism".

When you have time, please, get a look on my message about Brazil, before file shares expire.

[DTL]

I read about some concessions on using only 8x8 blocks, that's not optimal for HD, furthermore I would really like to see adaptive block sizes like H.265, a clear sky wouldn't need less than 32 block sizes whereas a city landscape would do better with 8. I will wait for stable.


The 8x8 blocks with AVX2 register file simply fastest for processing (full exa search with radius up to 3..3.5 (that is -4+3 for example). Because it can use H-stepping during search inside register file. Though as I see from practice - the search radiuses > 2 is still mostly experimental feature because default=2 and at finest level it even decreases to 1 for better speed. So it generally required to have only sp1 and sp2 search functions. sp3 and sp4 looks like rarely used by anyone. Though I have an idea that level 0 sp4 search will be faster in compare with 2levels 0,1 search with sp2 at level 1 and sp1 at level 0 - but it still an idea. The tests still not confirm it (though it were very few tests and with 8x8 only blocksize).

Though with latest speed test i try 4x 8x8 block search (in 4x1 arrangement that is faster to load from memory) with both H and V steps reload from memory (cache) and it still a bit faster (about 1.5x). Also the 4x 8x8 blocks may be arranged in 'macroblock' in different ways - like 4x1 in H direction or 2x2 in H and V and it makes 16x16 block. But this multi-8x8 block processing still have 8x8-precision refining because each 8x8 block is refined. In classic 16x16 block search there is no refining down to 8x8 granularity.

But because 16x16 block really fits in 1/2 of AVX2 register file so I will add 16x16 block search in 'standard' mode (with 16x16 granularity). It will not be as fast as 8x8 (inside search algoriphm) but should be still faster in compare with old 16x16 each pos SSE search (Expanding search). Because of 4times less vector data processing overhead in compare with 8x8 block size. Though it is still only for AVX2 capable chips. Sadly the war between intel and amd cause the amd to go to massive-multicore chips without avx512. So it looks the CPU chips manufacturers lost war with HW accelerators of large vectors processing and the future expansion of large vectors co-processors at standard CPU cores may be stopped. May be AVX2 is the last most commonly used large vector co-procession capability at general client CPU chips in current dying civilization. All other large vectors processing must be offloaded to separate HW accelerators (like GPU or other).
There some ideas of 'distributed' MAnalyse processing with sending 'workunits' to 'workers' that can be IP interconnected to any host in the net and also localhost can have any based 'worker' (also GPU-accelerated) to receive working task from main processing MAnalyse and return small data about vector data refined (12 bytes chunk per each block). But all workers must have access to the processing files (source). So users/programmers can make separated by IP layer 'vector task search/refine workers' with any processing capabilities (CPU/GPU/ASIC etc) and it will be easier to debug and scale. The size of 'workunit' may be as small as one block or as large as the tr-step (total tr-stepped frame for MDegrainN, though currently MDegrainN asks for separated each tr frame with GetFrame - not for all 2*tr MAnalyse data at once).

The AVX512 can fit 32x32 block for faster search but it looks the nowdays and close future of AVX512 in clients chips become not as nice as expected. Though some pro with highperformance workstations (some HP workstations have Xeons Gold with AVX512) and servers can use it.

I also move to the idea of dymanic block size processing because 8x8 still have lots of vector processing overhead. And if vectors of the local region are coherent enough - we can use multi-8x8 block processing at SAD search/refining with increasing of speed.

But typically clear sky and clear low-detailed (and low contrast ?) areas do not create enough coherent vectors fields. And as the most critical for speed level 0 search uses internally r=1 so we need really totally coherent prediction vectors x1=x2=x3=x3 and y1=y2=y3=y4) to use multi-block processing (or even arrange area to 4x macroblock) because any possible refining at finest level can be only to +-1 in x,y coordinates. May be maximum non-coherency may be +-1 because if predicted block coords are best - the search phase will simply confirm prediction and re-generate SAD value (as currently done with optPredictorType=3).

Or algoriphm of switching to 'macroblocks' need addition like not only check vectors coherency but also may be its spectum or total contrast, or something else.

[Boulder]

DGIndex and DGIndexNV both use the same random access logic, so if you can find a way to produce this error with DGIndex then perhaps pinterf will be able to look at it. We should chip in to get him an nVidia GPU, if anyone can find one.

Hmm, I could try and see if it occurs on MPEG2 sources. Somehow I've always thought this as a possible issue with NVidia drivers.

Then again, the strange seeking in SoftLimiter is also something worth investigating.

[Dogway]

Unfortunately none of them gave good results on movies such as Transformers, Pacific rim, etc... where the CGI sharpness was exasperated to increase "realism".

When you have time, please, get a look on my message about Brazil, before file shares expire.
I relaxed a bit for the last 2 or 3 days after the intense last week, but I haven't forgotten about your questions, thanks to it I improved ex_BM3D() which happens to have frame access issues with ffvideosource().
I already downloaded the clips and at first glance it seems to be a problem of high 'thSADC', but I need to give a deeper look at it.

As for softening I can't recommend anything outside deringing (which ex_luts("clamp") does) and/or blurring. I run tests on it and it worked fairly well for transformers with the antialiasing + deringing combo. Not sure what your goal is, this is as good as it gets without butchering the source.

Antialising for cutting out the sharpness
Deringing to eliminate acutance
Blurring (Optional) for extra softening

As a side effect mosquito noise also seems to be eliminated, so no bitrate hungry artifacts remain, specially if you pair it with a faint temporal denoiser like FluxSmooth or STpresso.
SantiagMod(strh=2,strv=2)
a=last
ex_luts(last,removegrain(12,0),mode="clamp",UV=1)

th = ex_bs(7, 8, bi, false)
ex_lutxy(last,a,Format("x y - abs {th} > y x y min ?")) # hard threshold

# The next very optional
msk=ex_makediff(ex_guidedblur(12,use_gauss=false,regulation_mode=2), aug=10, UV=128)
ex_merge(last,ex_boxblur(0.15,mode="weighted"),msk,luma=true)

Raw
http://i.imgur.com/EdfPh2zm.png (https://i.imgur.com/EdfPh2z.png)
AA+DR
http://i.imgur.com/jwXUyw1m.png (https://i.imgur.com/jwXUyw1.png)
AA+DR+Blur
http://i.imgur.com/QTDKMFkm.png (https://i.imgur.com/QTDKMFk.png)
AA+DR+Localized Blur
http://i.imgur.com/3pjn57gm.png (https://i.imgur.com/3pjn57g.png)

@Boulder: Can you try with Dyn=0? I think the issues may come from ShowChannels() or something, maybe ScriptClip() since I'm using "After_Frame=True". Until pinterf has a look on what is preventing ScriptClip to catch ShowChannels() globals I can't switch to the newer ScriptClip() syntax (more MT friendly)

[Boulder]

@Boulder: Can you try with Dyn=0? I think the issues may come from ShowChannels() or something, maybe ScriptClip() since I'm using "After_Frame=True". Until pinterf has a look on what is preventing ScriptClip to catch ShowChannels() globals I can't switch to the newer ScriptClip() syntax (more MT friendly)

Yes, dyn=0 works.

[tormento]

I saw you kept Spresso updated. Any news about STpresso? :)

[Dogway]

Just updated SPresso again with more HD modes. STpresso I haven't made it, but you can use STTWM() which is now included in ExTools.

I'm also in the middle of a big update for all my filters, big in the sense of importance not number of changes, since I'm going to rework the whole HBD scale thingy to reach parity with AVS+ core and masktools2, after the changes in test26 (https://forum.doom9.org/showthread.php?p=1957551#post1957551). This is important for MIX mods but I need to synchronize all the scripts to upload at once.

[tormento]

STpresso I haven't made it, but you can use STTWM() which is now included in ExTools.
Thanks, I've noticed your post some weeks ago.

Unfortunately I have absolutely no idea about how to translate the parameters from STpressto to STTWM.

[Dogway]

sthres -> limit
tthres -> tlimit
tw -> tbias
sw -> bias
aw -> back

Anyway, it's not a strict port but the rationale is the same, defaults are to resemble STPresso defaults. The difference is that I use weighted median for both spatial and temporal, and that the temporal part is that, a simple weighted temporal median while fluxsmoothT is a mix of temporal average and temporal median.
Also the temporal part is not motion compensated because a weighted temporal median is already self motion protected.

The benefit of all these changes is greater speed and simplicity. The only thing that can be improved is to use Didée style thresholding/limiting or my own soft threshold solution.

[tormento]

Anyway, it's not a strict port but the rationale is the same, defaults are to resemble STPresso defaults.
I have tried to apply your hints to my scripts, trying a really angry noise beast clip from 12 angry men movie (https://send.cm/d/6WES).

Despite being a BW movie, it has some really difficult trade between noise reduction and details (marble, clothes, wood, smoke), plus I am having really a hard time to obtain a decent bitrate when encoding.

I have tried the some of your previous hints, plus other I knew already, such as:

ConvertBits(16)
SMDegrain (tr=9, thSAD=900, refinemotion=true, contrasharp=false, PreFilter=5, plane=4, chroma=true)
fmtc_bitdepth (bits=8,dmode=8)

ConvertBits(16)
pre=smdegrain(tr=6,mode="temporalsoften",blksize=32,thSAD=900,prefilter=5,contrasharp=false,refinemotion=true)
smdegrain(tr=6,mode="MDegrain",blksize=32,prefilter=pre,thSAD=900,LFR=false,contrasharp=false,refinemotion=true)
fmtc_bitdepth (bits=8,dmode=8)

ConvertBits(16)
Spresso(limit=10,bias=30).STTWM(sw=30, tw=49, aw=1, sthres=10, tthres=8)
fmtc_bitdepth (bits=8,dmode=8)

ConvertBits(16)
Spresso(10,30).STpresso(limit=10,bias=30,RGmode=4,tthr=22,tlimit=8,tbias=49,back=1,mc=true)
SMDegrain (tr=6, thSAD=600, refinemotion=true, contrasharp=false, PreFilter=4, plane=4, chroma=true)
fmtc_bitdepth (bits=8,dmode=8)

ConvertBits(16)
Spresso(limit=10,bias=30).STTWM(sw=30, tw=49, aw=1, sthres=10, tthres=8)
SMDegrain (tr=6, thSAD=600, refinemotion=true, contrasharp=false, PreFilter=4, plane=4, chroma=true)
fmtc_bitdepth (bits=8,dmode=8)

Useless to say, I am not having good results, both in quality and in size terms.

Can you pull a rabbit out of the hat?

[Dogway]

Is there a special scene or frame you have issues with? I tested with the Carrie call (I use tr 1 and 2 and thSAD=400 for the 2nd call) and found it good enough.
Also keep in mind this is a BW film so plane=0 and chroma=false.

The film shows "fizz" grain. It's in between the Carrie example and something stronger like 300. I found that a blksize of 32 is good to avoid catching these kind of big grains, along refinemotion this is a good approach. Filters like SPresso or STTWM are not suited here because they only filter minimal change noise, these are more for DVDs or faint grain.
My general approach with strong grain is a rather strong prefilter (with LFR), and a rather not so strong filter without LFR.

Personally for my encodes I don't like the clean look because they either look psychovisually blurry or sometimes it shows the "screen" effect, like grain stuck over a glass, so I apply some artificial grain after denoising.

[tormento]

Is there a special scene or frame you have issues with? I tested with the Carrie call (I use tr 1 and 2 and thSAD=400 for the 2nd call) and found it good enough.
Also keep in mind this is a BW film so plane=0 and chroma=false.
The issue is the resulting bitrate, too high for a BW movie.

The best results are from SPresso+STPresso+SMDegrain but the speed is in the order of 0.5 fpsL too slow. I am eager to see your porting of STPresso if it could achieve better speeds and the same quality.

Plus, I have found that, strangely, if you apply plane=0 and chroma=false, you obtain a lower noise cleanup.

[Boulder]

The issue is the resulting bitrate, too high for a BW movie.

BW movies are often very grainy. Also, chroma information is a much smaller portion of data than luma.

[tormento]

BW movies are often very grainy.
Results vary by the "type" of grain, i.e. silver particle size, distribution, etc. Some BW movies are really easy to compress and you can achieve really high compression ratio.
Also, chroma information is a much smaller portion of data than luma.
The strange thing is that using plane=0 and chroma=false, lowers the noise reduction, even using Greyscale() to cleanup chroma completely. That's a thing I will never understand.

[Dogway]

The chroma plane which is plain grey will be considered for the SAD calculation so it lowers the overall SAD value, moreover when I use a scaleCSAD of 1 for HD sources. Chroma planes are always greyish so the difference is not much but it might be noticeable. You have to counteract this with a higher thSAD.

[tormento]

@dogway

Do you see anything wrong with

SMDegrain (tr=6,PreFilter=4,thSAD=900,contrasharp=false,refinemotion=true, plane=0, chroma=false, Globals=2)
SMDegrain (tr=6, thSAD=900,contrasharp=false,refinemotion=true, plane=0, chroma=false, Globals=1)

instead of using higher tr and thSAD?

Which choice is, quality wise, better?

[Dogway]

Ah, I noticed some issues, well Globals is something I haven't touched in a long time.
I recommend you for the time being to not use Globals. Other than that I see the output is very clean, I'm not accustomed to using such high values, maybe you can add some LFR recovery. Maybe the first pass can use blksize of 32 to speed up processing, but this depends on what details are being wiped or not, it depend on grain amount.

Using two passes is fine, it indirectly multiplies the effect for a fraction of the performance. (-6) -6 0 6 (6) = 24 frames for the current frame output.

[tormento]

Ah, I noticed some issues, well Globals is something I haven't touched in a long time.
Please :) I think that the possibility to calculate vectors only once is useful.
I recommend you for the time being to not use Globals.
So should I use 2 lines of SMDegrain without passing vectors?
Other than that I see the output is very clean
I just can see some issues right before scene change, where details get somehow lost.
I'm not accustomed to using such high values, maybe you can add some LFR recovery.
Perhaps you are not fond of old movies, where grain is not details but simply useless noise. I keep on battling with that.

Do you suggest me to put LFR in the first line, in the second or both?

12 angry men, which I sent you a clip of, is really a bloody mess. With x264 crf 20, I am sweating to keep bitrate unter 4000kb/s and for a BW movie it is really a lot.

[Dogway]

I already fixed the issue with Globals. There are already many changes for new SMDegrain, I was waiting to add RGB support and cumulative YPlaneMin/YPlaneMax to ex_retinex but I lost part of that work plus I was having issues with runtime variables readback.

I only need to finish an issue with UHDhalf prefiltering (switched position so prefiltering happens on dowscaled version) and I do the release. Next v3.4.0 is a good candidate for long stable release.

Running 2 passes is a good idea for several reasons, as you might know 2 mean averages approximates a binomial weight:

ex_boxblur(3,mode="weighted") <-> ex_boxblur(1,mode="mean").ex_boxblur(1,mode="mean")

This is called tent blur and in the context of motion vectors this is more performant. Besides you calculate new vectors from a cleaner source so its a finer refinement of MV.

For MDegrain this is like running a median twice, so it approximates a weighted median.
Together you perform a stronger denoising but with stronger center weighting, in other words better protection and speed than a single call with higher 'tr'.

With Globals this is not possible because you are reusing the "dirty" Globals of the first pass (the output Globals of the first prefiltering) and the only benefit is the weighted median. (In contrast using the first pass as a prefiltering only you do refine the MV but don't perform the weighted median)
Globals was designed when combining filters that use MV in rather clean sources, for example for temporal limiting, frame interpolation, etc.

I do like old films, actually I hate new digital recorded movies but lately I have been only developing rather than encoding. If you use such high thSAD or tr, LFR and DCTFlicker by extensions are a must. By Didée words grain is a high frequency only property but DCT encoding propagates this to low frequency. 'DCTFlicker' is not perfect so I recommend running both at first pass and lower thSAD and tr for the second pass to don't destroy what was recovered.

You can optimize x264 bitrate by using preset 'slower' or 'very slow'. Using 'zones' for the opening and credits. Also using a range limiter in avisynth. And a light blur where motion vectors fail (the 'mfilter' arg)

[tormento]

You can optimize x264 bitrate by using preset 'slower' or 'very slow'. Using 'zones' for the opening and credits. Also using a range limiter in avisynth. And a light blur where motion vectors fail (the 'mfilter' arg)
Thanks for your precious advices.

I am using slow preset as my CPU is the real bottleneck of my computer.

What about introducing a "pass" or "recursion" parameter, to internally apply the consecutive denoising? Perhaps it could lead to more speed and less roundings.

[Dogway]

Ok, I just uploaded a new version. It needs latest avisynth, test32 I think. By the way I included a new prefilter, ex_FluxSmoothST(), this is MinBlur + FluxSmoothT, which share the same concept but one is spatial and the other temporal. So now DFTTest is 4, KNLMeanCL 5 and BM3D 6.

I'm reworking the whole packs to be numerically stable and fast by using the latest updates by pinterf (we should build him a monument or at least a xmas present :D )

You might notice some speed improvements when I update ExTools, currently I'm finishing QC but it might take more days to be sure I don't screw it. I mainly have to assure that the new f32 scale_inputs works fine across the board along the new range_max constant (as full scale).

ex_retinex() needs more work to include cumulative stats, they will be in next version.

Do you mean recursion for MDegrain only (not MV)? That's possible. I will run tests to see how it improves things. It fits in the scope of current recursion solution by prepending an "r", so it would work as mode="rMDegrain".


EDIT: sneakingly added recursion to MDegrain, test it out to find if it fills your needs. I might give it a revision for the next version, as Didée was doing something similar back then.

[tormento]

So now DFTTest is 4, KNLMeanCL 5 and BM3D 6.
I'd prefer to have an added number than to change years of habits. :)
Do you mean recursion for MDegrain only (not MV)?
I meant a way to replicate 2 calls of SMDegrain consecutively but internally and optimized, as we were talking about the opportunity to have 2 calls with minor THSAD instead of 1 only with bigger one.

[tormento]

I will run tests to see how it improves things. It fits in the scope of current recursion solution by prepending an "r", so it would work as mode="rMDegrain".
I have looked at SMDegrain AVSI but I can't get how to make recursion work.

I can't see any parameter to add but only a check about the "r" letter on MDegrain and I can't understand how to make it work.

[Dogway]

Yes, declare mode="rMDegrain" in the call and it will use recursion, but only for the MDegrain part not the vectors. As for to replicate 2 consecutive calls, I think it's best to do it explicitly as currently is done, I prefer simple things that can be seen and there wouldn't be speed gain at all (you only save a single TV->PC range).

Bu the way I'm reworking ex_gaussianblur() (used in LFR and ex_retinex()) as it was designed with a single resolution size, but it changes depending on resolution so I had to make a multi-regression fit. The good part is that it will fit even better than before. I will release ExTools in 2 days.

[tormento]

As for to replicate 2 consecutive calls, I think it's best to do it explicitly as currently is done
I hoped to save some video memory, as the dual line recursion eats it like peanuts and I have to increase the number of threads: AVS doesn't like 2xSMDegrain so much and sits idle for the most of machine time.

[Dogway]

Well yes, that's an issue on mvtools, running manalyze several times has a great impact on performance, that's why among other things I don't like creating more MV in scripts like STPresso. DCFlicker uses it too, so it's a bit on the heavy size.

Back in the time I used to split the first pass of SMDegrain into a lossless file and then run the second call on this, this usually was faster than in a single script. MPPipeline also helped on this regard. Now for HD I don't see lossless being an option, maybe pseudo-lossless can cut it, a x264 profile at 12-bit with pseudo-lossless settings.

[tormento]

Now for HD I don't see lossless being an option, maybe pseudo-lossless can cut it, a x264 profile at 12-bit with pseudo-lossless settings.
x264 CRF 0 is of great use but unfortunately Nvidia can read 8 bit AVC 4:2:0 only, leaving me with the option to index it with some software only. HEVC is out of question, as it is too much CPU intensive.

[Dogway]

I have been making a SAD sampler for the last 2 days without success, so I will leave this here in case someone sees an issue.

# ex_Luma_Rebuild(s0=6)
# SAD_sampler(show=2)


## Dependencies:
## ExTools
## FredAverage
## ResizersPack
## Grade (for show)
## MasksPack (for show)
## TransformsPack (for show)

# show: 0: no debug, 1: show sampled area 2: show sampled area in diff mode
function SAD_sampler(clip src, int "show") {

show = Default(show, 0)

mmod(src,128,mode="crop")
b=ConvertBits(8, dither=-1).ExtractY()

w = width()
h = height()
davg=0 grey=0
id = 0 fr = 0
# x = 1 y = 1
OK = false

while (!OK) {

for (x = 1, w/128, 1) {

stridex = 128*x

for (y = 1, h/128, 1) {

stridey = 128*y

c = b.Crop(stridex-128,stridey-128,-w+stridex,-h+stridey)
a = c.FredAverage()
d = c.TemporalSoften(10,255,0,30,2)
# e = c.ex_edge("frei-chen",0,255)
diff = ex_makediff(a, d, aug=5, metric="none", UV=128)
# diff = ex_logic(diff, e, "max", UV=128)

ScriptClip(diff, function[a,d,x,y,c,show] () {

davg = (AverageLuma(last,-5)+AverageLuma(last)+AverageLuma(last,+5))/3.
grey = (AverageLuma(a, -5)+AverageLuma(a) +AverageLuma(a, +5))/3.
OK = (davg < 40.) && (90. < grey < 126.)
d = show > 1 ? last : show > 0 ? c : c
OK ? propSet(d,"_x_id",-1,0).propSet( "_y_id",-1,0).propSet("_diff",davg,0).propSet("_grey",grey,0) : \
propSet(d,"_x_id", x,0).propSet( "_y_id", y,0).propSet("_diff",davg,0).propSet("_grey",grey,0)
} )

yy = propGetInt("_y_id")
xx = propGetInt("_x_id")
EOL = (w+h)/128.
t1 = id/EOL == 1.0
t2 = id/EOL == 2.0
t3 = id/EOL == 3.0
t4 = id/EOL == 4.0
fr = t4 ? fr+24*3 : fr
b = t1 ? b.PadResize(w,h,biasw=64,biash=0) : \
t2 ? b.PadResize(w,h,biasw=0, biash=64) : \
t3 ? b.PadResize(w,h,biasw=-64,biash=0) : \
t4 ? b.PadResize(w,h,biasw=0,biash=-64).selectevery(1,fr) : b

if (show > 0 && yy<0) {
grey = round(propGetFloat("_grey"))
davg = round(propGetFloat("_diff"))
src = b.subtitle(Format("id:{id} fr:{fr}\nx:{x} y:{y}\ng:{grey} d:{davg}"),lsp=1)
bg = BoxMask(src,x*128-128,x*128,y*128-128,y*128,invert=false,matchformat=true)
ex_blend(src,bg,"multiply",opacity=0.3)
}
x = yy<0 ? w/128 : t1 || t2 || t3 || t4 ? 1 : xx
y = yy<0 ? h/128 : t1 || t2 || t3 || t4 ? 1 : yy
id = yy == h/128 ? id : id + 1
id = t4 ? 0 : id
OK = yy<0

}
}
OK ? last : Undefined()
}

I also tested adding the variables into an array to save on propSet/Get calls, but array items can't be dynamic it seems.
I will continue with last fixes for ExTools, SharpenersPack and SMDegrain.

[anton_foy]

Strange, what version of avs+ are you on? propNumElements() works for me outside runtime environments.

Same error here with LSFmod Im on Avs+ version r3577 (testbulid 34), error at line 541 in LSFmod-script.
EDIT: also smdegrain at line 61.

[tormento]

Trying ResizersPack-7.1~Dogway.avsi

Script error: Invalid arguments to function 'width'.
(D:/Programmi/Media/AviSynth+/plugins64/ResizersPack-7.1~Dogway.avsi, line 171)
(D:/Programmi/Media/AviSynth+/plugins64/ResizersPack-7.1~Dogway.avsi, line 150)
(D:/Programmi/Media/AviSynth+/plugins64/TransformsPack-1.0rc29~Dogway.avsi, line 136)
(D:/Programmi/Media/AviSynth+/plugins64/TransformsPack-1.0rc29~Dogway.avsi, line 145)
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.3.9d~Dogway.avsi, line 914)
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.3.9d~Dogway.avsi, line 917)
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.3.9d~Dogway.avsi, line 228)

[kedautinh12]

Trying ResizersPack-7.1~Dogway.avsi

Script error: Invalid arguments to function 'width'.
(D:/Programmi/Media/AviSynth+/plugins64/ResizersPack-7.1~Dogway.avsi, line 171)
(D:/Programmi/Media/AviSynth+/plugins64/ResizersPack-7.1~Dogway.avsi, line 150)
(D:/Programmi/Media/AviSynth+/plugins64/TransformsPack-1.0rc29~Dogway.avsi, line 136)
(D:/Programmi/Media/AviSynth+/plugins64/TransformsPack-1.0rc29~Dogway.avsi, line 145)
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.3.9d~Dogway.avsi, line 914)
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.3.9d~Dogway.avsi, line 917)
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.3.9d~Dogway.avsi, line 228)

Cause ResizersPack latest ver update for extools v6.9 but extools just updated v6.8

[tormento]

Cause ResizersPack latest ver update for extools v6.9 but extools just updated v6.8
Version 7.0 works perfectly fine and I think it was the first one prepared for newer extools. Only Dogway will tell us what happened :)

[Dogway]

Ok, I just updated ExTools so hope things go back to normal slowly. Still need to update SMDegrain. TransformsPack I don't know what the issue is?
I was holding back for the ex_bs() refactor but it was delayed on conversations with pinterf so decided to do it for next version, so sad v6.9 will be so short-lived :o

[anton_foy]

Ok, I just updated ExTools so hope things go back to normal slowly. Still need to update SMDegrain. TransformsPack I don't know what the issue is?
I was holding back for the ex_bs() refactor but it was delayed on conversations with pinterf so decided to do it for next version, so sad v6.9 will be so short-lived :o

Nice work! I replaced with new Extools but LSFmod get another error:
ex_lutspa does not have a named argument "fulls"

EDIT: sorry my bad, it was deblockpack but when I updated from 1.5 to 1.6 I get that runtime error again at line 98.

[Dogway]

LSFmod and TransformsPack still need to be updated. DeblockPack is updated so it should work, line 98? Do you mean this line?:
propGetInt (clp,"_ColorRange") == 0 : rgb

[Forteen88]

The issue is the resulting bitrate, too high for a BW movie.

The best results are from SPresso+STPresso+SMDegrain but the speed is in the order of 0.5 fpsL too slow. I am eager to see your porting of STPresso if it could achieve better speeds and the same quality.

Plus, I have found that, strangely, if you apply plane=0 and chroma=false, you obtain a lower noise cleanup.If I have to pick either SPresso or STPresso, would it be better to pick STPresso to encode DVD-videos, because it's temporal? To my understanding, SPresso is probably better to encode a video with a bunch of still-images.

[Boulder]

Video is a bunch of still images you know..

[Forteen88]

Video is a bunch of still images you know..:) I think that you understand what I mean.
STPresso for sequential video.

[Dogway]

I always favour some temporal filtering, so you can try STPresso. It has bias options between spatial and temporal, it's a tad slower than SPresso so for a faster alternative I made STTWM(). You can try them and use what suits your needs.

[anton_foy]

LSFmod and TransformsPack still need to be updated. DeblockPack is updated so it should work, line 98? Do you mean this line?:
propGetInt (clp,"_ColorRange") == 0 : rgb

It says "propnumelements: this filter can only be used within run-time filters. DeblockPack.avsi, line 98"

[Forteen88]

I always favour some temporal filtering, so you can try STPresso....Thanks.

[Dogway]

It says "propnumelements: this filter can only be used within run-time filters. DeblockPack.avsi, line 98"

I think you don't have latest AVS+ installed because it's working fine here. frameprops can be read from frame index 0 since a few versions back.


By the way, I just uploaded a refactor of FlatMask() inspired by the recent thread here (https://forum.doom9.org/showthread.php?t=183533)on edge masks for cartoons.
I remade it entirely with ExTools and while it isn't exactly the same it probably works faster than doing full segmentation like described in the paper.
It serves two purposes, edge masks without double edges, and if fine tuned also as a flat mask. It uses an internal limiter so it also can be used to mix filtered clips to pass only on flat areas.

http://i.imgur.com/qLkLLbem.png (https://i.imgur.com/qLkLLbe.png)

http://i.imgur.com/45OtuXlm.png (https://i.imgur.com/45OtuXl.png)

[Shinkiro]

LSFmod broken
https://i.imgur.com/pho0VTB.png

[Dogway]

Yes, updating soon.

[anton_foy]

I think you don't have latest AVS+ installed because it's working fine here. frameprops can be read from frame index 0 since a few versions back.


Don't know what Im doing wrong I replaced all avisynth.dll's with the newest "Avisynth+ 3.7.1 test build 34 (20211208)"
In Avisynth_repository>x86-folder and in System32-folder.

EDIT: BTW. Thanks Dogway for the applygradientcurves substitution!!!

[Dogway]

Are you using avisynth x86? If not files should go into the x64 folder and system32. In other words in the test34 zip go to the x64 folder and copy Avisynth.dll and DeVil.dll into System32, then the files from plugins folder into "plugins64+" folder of your avisynth+ installation. If this doesn't work you might need to check your paths with avsmeter "avsmeter64 avsinfo -lf >AVSinfo.log"

This is how it looks here:
VersionString: AviSynth+ 3.7.1 (r3577, master, x86_64)
VersionNumber: 2.60
File / Product version: 3.7.1.0 / 3.7.1.0
Interface Version: 8
Multi-threading support: Yes
Avisynth.dll location: C:\Windows\system32\avisynth.dll
Avisynth.dll time stamp: 2021-12-08, 19:27:35 (UTC)
PluginDir2_5 (HKLM, x64): C:\Program Files (x86)\AviSynth+\plugins64
PluginDir+ (HKLM, x64): C:\Program Files (x86)\AviSynth+\plugins64+

[anton_foy]

Are you using avisynth x86? If not files should go into the x64 folder and system32. In other words in the test34 zip go to the x64 folder and copy Avisynth.dll and DeVil.dll into System32, then the files from plugins folder into "plugins64+" folder of your avisynth+ installation. If this doesn't work you might need to check your paths with avsmeter "avsmeter64 avsinfo -lf >AVSinfo.log"

This is how it looks here:
VersionString: AviSynth+ 3.7.1 (r3577, master, x86_64)
VersionNumber: 2.60
File / Product version: 3.7.1.0 / 3.7.1.0
Interface Version: 8
Multi-threading support: Yes
Avisynth.dll location: C:\Windows\system32\avisynth.dll
Avisynth.dll time stamp: 2021-12-08, 19:27:35 (UTC)
PluginDir2_5 (HKLM, x64): C:\Program Files (x86)\AviSynth+\plugins64
PluginDir+ (HKLM, x64): C:\Program Files (x86)\AviSynth+\plugins64+

Yes I use 32-bit AVS+

[DTL]

I see default overlap param in SMDegrain = half block size. Do it really makes great visible quality benefit over the smaller size (like 2) ? I see enabling overlap (overlap > 0) in MDegrain significantly affect speed. With block size 8x8 overlap =4 may be 4 times slower of no-overlap. Overlap=2 runs about 60% of no-overlap speed. The overlap=0 really produces visible blocks edges (and make best speed but lowest quality).

Wiki lists:
overlap int = blksize/2
Must be *even* and *less* than block size. Common values: blksize/4 or blksize/2. Larger overlap looks slightly better and runs slower.

May be it exist some statistics what is most commonly used overlap value ?

Assuming with hardware motion search the MAnalyse will be *infinitely* fast - the overlapping processing of MDegrain on CPU will be the main speed limiter.

[Dogway]

Guess I didn't notice the change of defaults, typically settings are as in MVTools defaults so a SMDegrain dumb call is practically the same.

The documentation says:
"The greater overlap, the more blocks number, and the lesser the processing speed. However the default value 0 may cause blocking-like artefacts"

But I can test with blksize/4, I will have a look at zorr statistics but I also have pending my own statistics. For SMDegrain is that and automatic thSAD detection, but it will be in form of a helper function to be called explicitly. I think a wise strategy is to default to blksize/4 if refinemotion is used.

[zorr]

But I can test with blksize/4, I will have a look at zorr statistics but I also have pending my own statistics.

In my tests so far larger overlap most of the time means better quality. Some of my results can be found in this post (https://forum.doom9.org/showpost.php?p=1945684&postcount=31).

The test was using MFlowFPS while MCompensate would most likely be closer to what MDegrain is doing. Nevertheless I think the results also apply to MDegrain to certain degree at least.

Optimal overlap and overlapv were different in that test, I think because motion was almost perfectly horizontal. In the general case that doesn't apply so using blksize/2 is the best option. I have even suggested that MVTools allowed larger than blksize/2 overlaps since there's a clear trend of better quality the larger the overlap.

It would be interesting to test if the quality drop of having smaller overlap like blksize/4 can be compensated with other parameters which don't slow down the processing that much.

[kedautinh12]

Hi, sorry for question isn't relate post. I want prefilter convertoyuv444 but meet error: "both clip should same colorspace". I want to know what code can auto convert comeback original colorspace input clip?? Dogway

[Dogway]

What filter? SMDegrain? I tested and didn't get an issue. Post your call.


EDIT: By the way, happy not-so-happy day. Finally managed to craft a proper "optimized" sorting network for median7!! 6 months in the making finally. Well, actually it only saved 2 'max', 2 'dup1' and little else. Smart positioning of the lines has a greater impact on performance/multithreading. In any case speed is same as before since I have been feeling a slight slowdown of AVS+ from test20s.

[kedautinh12]

Oh, i asked a question for a code can convert comeback original colorspace based your prefilter but i know how to do that, thanks. Btw, i used to want give you a private message but your inbox full and i decide comment in here :D

[Dogway]

Yes, it's full many years 'round here. I have very "interesting" PM so I don't want to delete them ^^
I made MatchClip() to match format from clip 'a' to 'b'. In fast mode (default) it should work fine, my goal is to finish ConvertFormat() and default to it at some point.
You also have MatchColorFormat() from Raffriff42, and yesterday discovered HistogramRGB.avsi where it includes ConvertToMatch().

[StainlessS]

Yes, it's full many years 'round here. I have very "interesting" PM so I don't want to delete them ^^
You can save your PM's (to a file, I think - I've never done it).

[And just think of the very interesting PM's that you never got]

[kedautinh12]

Yes, it's full many years 'round here. I have very "interesting" PM so I don't want to delete them ^^
I made MatchClip() to match format from clip 'a' to 'b'. In fast mode (default) it should work fine, my goal is to finish ConvertFormat() and default to it at some point.
You also have MatchColorFormat() from Raffriff42, and yesterday discovered HistogramRGB.avsi where it includes ConvertToMatch().

And can you give me any posible contact when your massage full??:D

[guest]

Yes, it's full many years 'round here. I have very "interesting" PM so I don't want to delete them ^^
I made MatchClip() to match format from clip 'a' to 'b'. In fast mode (default) it should work fine, my goal is to finish ConvertFormat() and default to it at some point.
You also have MatchColorFormat() from Raffriff42, and yesterday discovered HistogramRGB.avsi where it includes ConvertToMatch().

Hi Dogway,

Yes, you can indeed save your PM's (all at once), to either a .csv, .xml or .txt

Go to your PM page, select your PM's (top right hand corner, check box), then bottom right hand corner "Selected Messages", drop down to show option menu.

Then you can start again, with new "interesting" PM, as StainlessS mentioned.

[kedautinh12]

I met error with match clip:
ConvertToYV411: only 8 bit sources allowed
(C:/Program Files (x86)/AviSynth+/plugins64+/TransformsPack.avsi, line 144)
(C:/Program Files (x86)/AviSynth+/plugins64+/TransformsPack.avsi, line 145)
(C:/Program Files (x86)/AviSynth+/plugins64+/TemporalDegrain-v2.4.1.avsi, line 440)
(C:/Program Files (x86)/AviSynth+/plugins64+/TemporalDegrain-v2.4.1.avsi, line 345)
(C:\Megui\AVSMeter3090\td-002.mkv.avs, line 4)

can you tell me any error with this function??
function DT_BM3D(clip a, float "sigma", int "radius", bool "CUDA", bool "chroma", int "device_id") {

s = Default(sigma, 3)
r = Default(radius, 1)
cd = Default(CUDA, true)
chr = Default(chroma, false)
device_id = Default(device_id, 0)
r = min(r,3) # Buggy when r > 3
a
ch = ((chr == true) && !Is444(a))
ch ? ConverttoYUV444(chromaresample="spline16") : last
cd ? BM3D_CUDA(sigma=s, radius=r, chroma=chr, fast=true, extractor_exp=6, device_id=device_id) : \
BM3D_CPU (sigma=s, radius=r, chroma=chr)
BM3D_VAggregate(radius=r)
ConvertBits(16, dither=1)
ch ? MatchClip(a) : mergechroma(a)
}

[Dogway]

Thx, yes PMs when Doom9 was an hostile place. I like to keep a record of internet beefs as fun facts ^^
I will do backup then to start anew for the new year.

@kedautinh12: Are you testing with 411 clip? AVS+ doesn't support HBD 411, and BM3D doesn't support anything less than 32-bit.

You can fix the function with ConvertBits(BitsPerComponent(a)) before MatchClip(a), I need to implement range and bitdepth matching into MatchClip() (already done), will update soon.

[kedautinh12]

Thx, yes PMs when Doom9 was an hostile place. I like to keep a record of internet beefs as fun facts ^^
I will do backup then to start anew for the new year.

@kedautinh12: Are you testing with 411 clip? AVS+ doesn't support HBD 411, and BM3D doesn't support anything less than 32-bit.

You can fix the function with ConvertBits(BitsPerComponent(a)) before MatchClip(a), I need to implement range and bitdepth matching into MatchClip() (already done), will update soon.

I'm testing with 420 clip

edit: i added ConvertBits(BitsPerComponent(a)) but error still exist. I don't know why my testing clip is 420 but when use with MatchClip(a), it's become 411

[Dogway]

Can you show a small script showing the problem?

[kedautinh12]

script showing problem
ConvertToYV411: only 8 bit sources allowed
(C:/Program Files (x86)/AviSynth+/plugins64+/TransformsPack.avsi, line 144)
(C:/Program Files (x86)/AviSynth+/plugins64+/TransformsPack.avsi, line 145)
(C:/Program Files (x86)/AviSynth+/plugins64+/TemporalDegrain-v2.4.1.avsi, line 442)
(C:/Program Files (x86)/AviSynth+/plugins64+/TemporalDegrain-v2.4.1.avsi, line 345)
(C:\Megui\AVSMeter3090\td-002.mkv.avs, line 4)


here my script
function DT_BM3D(clip a, float "sigma", int "radius", bool "CUDA", bool "chroma", int "device_id") {

bi = BitsPerComponent(a)
s = Default(sigma, 3)
r = Default(radius, 1)
cd = Default(CUDA, true)
chr = Default(chroma, false)
devID = Default(device_id, 0)
r = min(r,3) # Buggy when r > 3
a
ch = ((chr == true) && !Is444(a))
ch ? ConverttoYUV444(chromaresample="spline16") : last
cd ? BM3D_CUDA(sigma=s, radius=r, chroma=chr, fast=true, extractor_exp=6, device_id=devID) : \
BM3D_CPU (sigma=s, radius=r, chroma=chr)
BM3D_VAggregate(radius=r)
ConvertBits(16, dither=1)
ConvertBits(bi)
ch ? MatchClip(a) : mergechroma(a)
}

[Dogway]

That's not a script but a function declaration. Anyway I updated TransformsPack (https://github.com/Dogway/Avisynth-Scripts/blob/master/TransformsPack.avsi)so check if it fixes your problem.

I'm going to rest for a month or so now, I will still be uploading updates but less frequently.

[kedautinh12]

i updated and still error. my script:
SetFilterMTMode("DEFAULT_MT_MODE", MT_MULTI_INSTANCE)
LoadPlugin("C:\Megui\MeGUI-2924-64\tools\lsmash\LSMASHSource.dll")
LWLibavVideoSource("C:\Megui\td-002.mkv")
TemporalDegrain2(postFFT=5)


TemporalDegrain test ver:
https://drive.google.com/file/d/1V3xvAfh5LFdBy0I8N8TAKcws562_m3au/view?usp=sharing

[Dogway]

It wasn't consistent with bitdepth. You converted to 32-bit before DT_BM3D, so afterwards DT_ContraSharpening was handling one clip in 16-bit and another in 32-bit.



#---------------------------------------
# post FFT

fullClip = dgNR2

fullSuper = (postTR > 0) ? fullClip.MSuper( pel=subpel, levels=1, chroma=ChromaNoise ) : NOP()
noiseWindow = (postTR == 0) ? fullClip : \
fullClip.MCompensate( fullSuper, vmulti.SelectRangeEvery(maxTR*2,postTR*2), tr=postTR, center=true, thSAD=thSAD2, thSCD1=thSCD1, thSCD2=thSCD2 )

dftDither = (postDither < 0) ? postDither*(-1) : 0
postDither = (postFFT == 5 && postDither < 1) ? 1 : postDither # ensure to ConvertBits down after BM3D
dnWindow = (postFFT == 0) ? RemoveGrain( postDither > 1 ? noiseWindow.ConvertBits(16) : noiseWindow, mode=1) : \
(postFFT == 1) ? neo_fft3d ( postDither > 0 ? noiseWindow.ConvertBits(16) : noiseWindow, y=LumaNoise?3:2, u=ChromaNoise?3:2, v=ChromaNoise?3:2, sigma=postSigma, bt=postTD, ncpu=fftThreads ) : \
(postFFT == 11)? FFT3DFilter( postDither > 0 ? noiseWindow.ConvertBits(16) : noiseWindow, plane=degrainPlane, sigma=postSigma, bt=postTD, ncpu=fftThreads ) : \
(postFFT == 2) ? FFT3DGPU( postDither > 0 ? noiseWindow.ConvertBits(16) : noiseWindow, plane=degrainPlane, sigma=postSigma*2/3, bt=postTD, precision=2, mode=1 ) : \
(postFFT == 3) ? neo_dfttest( postDither > 0 ? noiseWindow.ConvertBits(16) : noiseWindow, y=LumaNoise?3:2, u=ChromaNoise?3:2, v=ChromaNoise?3:2, sigma=postSigma*4, tbsize=postTD, dither=dftDither, threads=fftThreads ) : \
(postFFT == 13)? dfttest( postDither > 0 ? noiseWindow.ConvertBits(16) : noiseWindow, Y=LumaNoise, U=ChromaNoise, V=ChromaNoise, sigma=postSigma*4, tbsize=postTD, threads=fftThreads, dither=dftDither ) : \
(postFFT == 4) ? DT_KNLMeansCL( postDither > 0 ? noiseWindow.ConvertBits(16) : noiseWindow, a=2, d=postTR, h=postSigma, Luma = LumaNoise, Chroma = ChromaNoise, device_type="GPU", device_id=devId) : \
(postFFT == 5) ? DT_BM3D( postDither > 0 ? noiseWindow.ConvertBits(16) : noiseWindow, radius=postTR, sigma=postSigma, CUDA=cuda, chroma=ChromaNoise, device_id=devId ) : \
(postFFT == -1)? HQDn3D( postDither > 0 ? noiseWindow.ConvertBits(16) : noiseWindow, 0,0,4,1, u=ChromaNoise?3:2, v=ChromaNoise?3:2) : NOP()
dnWindow = dnWindow.SelectEvery( postTD, postTR )
denoised = (postDither == 1) ? dnWindow.ConvertBits(outputBits,dither=1) : dnWindow

csOrg = (postDither == 2) ? dgO.ConvertBits(16) : dgO
sharpened = DT_ContraSharpening(denoised, csOrg, extraSharp)
sharpened = (postDither == 2) ? sharpened.ConvertBits(outputBits,dither=1) : sharpened

# mix with original to leave the posibility to decrease the denoising strength
sharpened = postMix > 0 ? mt_lutxy(dgO,sharpened,"x "+String(postMix)+" * y "+String(int(100-postMix))+" * + 100 /") : sharpened

# Crop off temporary vertical padding
cropped = sharpened

output = debug ? Select(outputStage, dgNR1x, dgNR2, cropped, repair0, spatialBlur, srchClip, dgLimit, denoised) : Select(outputStage, dgNR1x, dgNR2, cropped)
return output
}


function DT_BM3D(clip a, float "sigma", int "radius", bool "CUDA", bool "chroma", int "device_id") {

bi = BitsPerComponent(a)
s = Default(sigma, 3)
r = Default(radius, 1)
cd = Default(CUDA, true)
chr = Default(chroma, false)
devID = Default(device_id, 0)
r = min(r,3) # Buggy when r > 3
a
ch = ((chr == true) && !Is444(a))
ch ? ConverttoYUV444(chromaresample="spline16").ConvertBits(32) : ConvertBits(32)
cd ? BM3D_CUDA(sigma=s, radius=r, chroma=chr, fast=true, extractor_exp=6, device_id=devID) : \
BM3D_CPU (sigma=s, radius=r, chroma=chr)
BM3D_VAggregate(radius=r)
ConvertBits(bi, dither=1)
ch ? MatchClip(a) : mergechroma(a)
}

[Dogway]

By the way, last night I had an enlightenment and today I refactored ex_vibrance(). Fast, HBD support and straight to the point. 'Recover' mode also works now to recover lost saturation by filtering, derainbows, etc. I plan to revisit SkinQualifier soon.

[anton_foy]

By the way, last night I had an enlightenment and today I refactored ex_vibrance(). Fast, HBD support and straight to the point. 'Recover' mode also works now to recover lost saturation by filtering

That sounds amazing just what I need!

[Dogway]

Yes, I remember when you needed something similar a few months back. Recover mode never was implemented so this is some achievement, actually it was very easy, just divide the masks but I wasn't so math savvy then.

[Fjord]

Fantastic collection of functions with references!
Just browsing your code - noted a string with trailing space "CAM16 " at line 2822 in function Format_fuzzy_search(...)
...
cm=
...
\ : (FindStr(model, "CAM16" )>0) ? ["CAM16 ", "YUV",34]

I assume the trailing space is unintentional.

[Dogway]

Yes, thanks! Anyway many models aren't implemented yet, CAM16 is said to be complex while not being so uniform as recent ones. TransformsPack (namely ConvertFormat()) is a long term project, the rest are pretty mature.

I updated GradePack with some optimizations for ex_vibrance() and a new filter, Skin_Qualifier() to create skin masks. It works like raffriff42's SmoothHueMask() with different false positives, but generally less amount of them. It's also slower since it's processed in HBD internally and a conversion to OkLab is performed for perceptually uniform HUE. Runs at about 60fps on my CPU -Prefetch(8)-

I'm going now to implement HSVxHSV, that is; HUExHUE, HUExSAT, SATxLUM, etc.

[anton_foy]

Yes, I remember when you needed something similar a few months back. Recover mode never was implemented so this is some achievement, actually it was very easy, just divide the masks but I wasn't so math savvy then.

Although I am not able to try it yet, it is mentioned in the description that it needs the info from the source clip and filtered clip(-rc/reference clip) right?
So if I want it to recover saturation after I have denoised/filtered the clip how do I do that? Maybe stupid question but I thought if I call ex_vibrance(md="recover",rc=filtered) after filtering/denoise the source is then already filtered? Or should I put the script like this:
a=last
Filtered=*some denoiser*
ex_vibrance(md="recover",rc=filtered)?

EDIT: ah stupid of me! I see now that the rc-clip is the source and a=filtered of course. Forget what I just wrote.

[Dogway]

Yes that's right, rc is the reference clip (before being filtered). I think this will be very useful specially with fft3d which smears chroma a lot, but at the same time is very good for derainbow.

[anton_foy]

Yes that's right, rc is the reference clip (before being filtered). I think this will be very useful specially with fft3d which smears chroma a lot, but at the same time is very good for derainbow.

Yes precisely this with fft3d I refered to in a previous thread but I think my explanation was confusing. I would like to understand the "mechanism" behind your recover-mode, maybe the same concept could apply for sharpening and other things if modified. But I think your math knowledge will go far over my head for me to understand it.

[Dogway]

Actually it's the same concept as sharpening. For sharpening you subtract a blurred version with the original, if you remove the low frequency (blurred) to an image you are left with the high frequencies, add this high frequency to the original and that's sharpening.

For saturation (recovery mode) it's the same but this time I used division and multiplication, don't know why, maybe it also works with subtract and sum. Basically get the saturation mask of the original and filtered, do a difference and add back to the filtered.

Maths are easy if explained well I think. I'm buying some math books this xmas, would like to get into NN and Julia in the future but I always find myself busy with some other stuff.

[kedautinh12]

It wasn't consistent with bitdepth. You converted to 32-bit before DT_BM3D, so afterwards DT_ContraSharpening was handling one clip in 16-bit and another in 32-bit.



#---------------------------------------
# post FFT

fullClip = dgNR2

fullSuper = (postTR > 0) ? fullClip.MSuper( pel=subpel, levels=1, chroma=ChromaNoise ) : NOP()
noiseWindow = (postTR == 0) ? fullClip : \
fullClip.MCompensate( fullSuper, vmulti.SelectRangeEvery(maxTR*2,postTR*2), tr=postTR, center=true, thSAD=thSAD2, thSCD1=thSCD1, thSCD2=thSCD2 )

dftDither = (postDither < 0) ? postDither*(-1) : 0
postDither = (postFFT == 5 && postDither < 1) ? 1 : postDither # ensure to ConvertBits down after BM3D
dnWindow = (postFFT == 0) ? RemoveGrain( postDither > 1 ? noiseWindow.ConvertBits(16) : noiseWindow, mode=1) : \
(postFFT == 1) ? neo_fft3d ( postDither > 0 ? noiseWindow.ConvertBits(16) : noiseWindow, y=LumaNoise?3:2, u=ChromaNoise?3:2, v=ChromaNoise?3:2, sigma=postSigma, bt=postTD, ncpu=fftThreads ) : \
(postFFT == 11)? FFT3DFilter( postDither > 0 ? noiseWindow.ConvertBits(16) : noiseWindow, plane=degrainPlane, sigma=postSigma, bt=postTD, ncpu=fftThreads ) : \
(postFFT == 2) ? FFT3DGPU( postDither > 0 ? noiseWindow.ConvertBits(16) : noiseWindow, plane=degrainPlane, sigma=postSigma*2/3, bt=postTD, precision=2, mode=1 ) : \
(postFFT == 3) ? neo_dfttest( postDither > 0 ? noiseWindow.ConvertBits(16) : noiseWindow, y=LumaNoise?3:2, u=ChromaNoise?3:2, v=ChromaNoise?3:2, sigma=postSigma*4, tbsize=postTD, dither=dftDither, threads=fftThreads ) : \
(postFFT == 13)? dfttest( postDither > 0 ? noiseWindow.ConvertBits(16) : noiseWindow, Y=LumaNoise, U=ChromaNoise, V=ChromaNoise, sigma=postSigma*4, tbsize=postTD, threads=fftThreads, dither=dftDither ) : \
(postFFT == 4) ? DT_KNLMeansCL( postDither > 0 ? noiseWindow.ConvertBits(16) : noiseWindow, a=2, d=postTR, h=postSigma, Luma = LumaNoise, Chroma = ChromaNoise, device_type="GPU", device_id=devId) : \
(postFFT == 5) ? DT_BM3D( postDither > 0 ? noiseWindow.ConvertBits(16) : noiseWindow, radius=postTR, sigma=postSigma, CUDA=cuda, chroma=ChromaNoise, device_id=devId ) : \
(postFFT == -1)? HQDn3D( postDither > 0 ? noiseWindow.ConvertBits(16) : noiseWindow, 0,0,4,1, u=ChromaNoise?3:2, v=ChromaNoise?3:2) : NOP()
dnWindow = dnWindow.SelectEvery( postTD, postTR )
denoised = (postDither == 1) ? dnWindow.ConvertBits(outputBits,dither=1) : dnWindow

csOrg = (postDither == 2) ? dgO.ConvertBits(16) : dgO
sharpened = DT_ContraSharpening(denoised, csOrg, extraSharp)
sharpened = (postDither == 2) ? sharpened.ConvertBits(outputBits,dither=1) : sharpened

# mix with original to leave the posibility to decrease the denoising strength
sharpened = postMix > 0 ? mt_lutxy(dgO,sharpened,"x "+String(postMix)+" * y "+String(int(100-postMix))+" * + 100 /") : sharpened

# Crop off temporary vertical padding
cropped = sharpened

output = debug ? Select(outputStage, dgNR1x, dgNR2, cropped, repair0, spatialBlur, srchClip, dgLimit, denoised) : Select(outputStage, dgNR1x, dgNR2, cropped)
return output
}


function DT_BM3D(clip a, float "sigma", int "radius", bool "CUDA", bool "chroma", int "device_id") {

bi = BitsPerComponent(a)
s = Default(sigma, 3)
r = Default(radius, 1)
cd = Default(CUDA, true)
chr = Default(chroma, false)
devID = Default(device_id, 0)
r = min(r,3) # Buggy when r > 3
a
ch = ((chr == true) && !Is444(a))
ch ? ConverttoYUV444(chromaresample="spline16").ConvertBits(32) : ConvertBits(32)
cd ? BM3D_CUDA(sigma=s, radius=r, chroma=chr, fast=true, extractor_exp=6, device_id=devID) : \
BM3D_CPU (sigma=s, radius=r, chroma=chr)
BM3D_VAggregate(radius=r)
ConvertBits(bi, dither=1)
ch ? MatchClip(a) : mergechroma(a)
}

If change same you. Problem still exist

[Dogway]

It worked for me, with the same call.

By the way, I uploaded a YUV to RGB GamutWarning() filter.

[kedautinh12]

It worked for me, with the same call.

By the way, I uploaded a YUV to RGB GamutWarning() filter.

Can you check with this video??
https://drive.google.com/file/d/1wSLEl0lAhsvoxcfCNczeZOSRTkoFYfAB/view

[Dogway]

Just tested and works just fine. Maybe test with an older AvspMod. I haven't updated yet to latest because didn't have time to go one by one updating function definitions.
Another thing might be plugin version (?). In these cases I empty the plugin folder with the minimum necessary and updated versions of scripts+plugins.

Is still the same 411 error? Sounds strange to me.

Another option for debugging is delete all the function outputs starting from the bottom, until you don't get an exception error, so you know where it fails.

setmemorymax(2048)
ffvideosource("...Desktop\td-002.mkv")
TemporalDegrain2(postFFT=5,debug=false)

[DTL]

With some progress on DX12_ME it going to the question: Do SAD output from MAnalyse is (always) required for large hand-writed scripts using mvtools ? May be for some intermediate processing only x,y MVs components are enough ?

Currently I think calculating SAD for received from HWAcc MVs field takes about same time as MVs data path to and from HWAcc. So I think to make 'SAD-less' option for MAnalyse with DX12_ME search option.

With MDegrainN calculating SAD in MAnalyse makes processing slower because twice memory read for SAD in MAnalyse and for blocks blending in MDegrainN. So for faster still 2-filters based MDegrainN+MAnalyse it is faster to make option to MAnalyse do not calculate SAD and make MDegrainN signalling to calculate SAD before weights calculating.

As MAnalyse is core engine for MVtools to it looks nice to have some DX12_ME support but for much faster MDegrainN it is planned to make in the future total-in-HWAcc processing so much fewer options for MDegrainN will be (and no MAnalyse at all). Current Windows API for DX12_ME only support selecting block size of 8x8 or 16x16. Still do not test if 16x16 is faster (may be). Disabling load of chroma planes will possilby make very few speed gain because in YV12 they are small. The HW_ME always use chroma internally and 4:2:0 only. Also the pel is fixed to quater that is nice for quality but very slow in outer mvtools data flow. So for speed it is currently truncated to 1 (full pel).

Need testers for quality of currently available via Microsoft API ME - is it more or less usable and how compared with MAnalyse CPU-based search algoriphms. There is an ideas that NDIVIA-API ME (Optical Flow - latest generations) will be better. May be more settings for quality-speed balance avaialble. But it is limited to only NVIDIA hardware and still not implemented in AVS plugin.

Current testbuild and instructions in https://forum.doom9.org/showthread.php?p=1960271#post1960271

[Dogway]

Yes, some tests would be needed to ensure MDegrainN-only-SAD is enough to keep quality, I'm not sure of the implications but calculating twice sounds like redundant (?), maybe testing with Zopti. Ideally one single client should be enough (ie. MDegrainN), currently the problem was that MDegrainN was slower than MDegrainX but integrating everything into MDegrainN and fixing its speed issues should make up for it.

Unfortunately I'm still on Win7 so can't test. I would like to test some GPU AI based optical flow filter in AVS+. My GTX 1070 also only supports Video Codec SDK 7.0+ (not even Optical Flow SDK 1.0) of NVIDIA-API ME. Been wanting to buy an Ampere at a reasonably price for a year because there are no more drivers for Win7 since August.

The field that covers this is "optimal control", I'm still studying statistics and probability but maybe in a future I can dive into this and Rust programming.

Have been busy today with HSVxHSV(), almost but not there yet, maybe tomorrow. Then Harris corner, and then I will have a look at the SAD analysis code block. The thing is that I'm calculating SAD in the spatial domain and deriving this to MVTools which calculates SAD in temporal domain so spatial always has preference over temporal at least for first pass and refined later in a second pass.

EDIT: and Happy New Year everyone! Gorgeous year 2021 for AVS+.

[kedautinh12]

Just tested and works just fine. Maybe test with an older AvspMod. I haven't updated yet to latest because didn't have time to go one by one updating function definitions.
Another thing might be plugin version (?). In these cases I empty the plugin folder with the minimum necessary and updated versions of scripts+plugins.

Is still the same 411 error? Sounds strange to me.

Another option for debugging is delete all the function outputs starting from the bottom, until you don't get an exception error, so you know where it fails.

setmemorymax(2048)
ffvideosource("...Desktop\td-002.mkv")
TemporalDegrain2(postFFT=5,debug=false)

it's still 411 error. I don't know why

edit: it's only work if i used script
LoadPlugin("C:\Megui\MeGUI-2924-64\tools\ffms\ffms2.dll")
FFVideoSource("C:\Megui\td-002.mkv")
Converttoyuv444()
TemporalDegrain2(postFFT=5)
prefetch(2)

[DTL]

" calculating twice sounds like redundant (?)"

Not twice but in one of two places - either in MAnalyse or in MDegrain. I hope it can be auto-signaled from MAnalyse to MDegrain via current control path if the output stream from MAnalyse have valid SAD data or not. So for script writers it is easier - not need to set params in both MAnalyse and MDegrain about stream format inbetween - having valid SAD data or not.

Current speed problem of multi-frame ('temporal') denoising - too large 'internal' memory traffic.

Outside traffic for processing N input frames to output N frames is N-frames read and write to host memory.

In 'classic MVtools':
1. MAnalyse MVs search based on SAD - (2 * tr ) src+ref pairs = it is 4 * tr memory read ops
2. MDegrain (no-overlap) frames blending - (2 * tr + 1) memory read ops
Total is 4 * tr + (2 * tr + 1) = 6 * tr+1 memory read frames per 1 output frame. For tr 10..20 (0.5..1 seconds at 25 fps) it is 60..120 memory read frames for 1 output frame.

If enabling overlap MDegrain processing - it makes about 2..4x more memory read traffic at MDegrain processing stage and much slower processing. So I currently making no-overlap 'interpolated-weighting' blending mode for MDegrainN.

In current testbuild:
1. DX12_ME read (convert to NV12) for uploading is (2 * tr ) ref + 1 source = (2 * tr + 1)
2. MAnalyse SAD calculation (2 * tr) src+ref pairs = 4 * tr
3. MDegrain (no-overlap) frames blending - (2 * tr + 1) memory read ops
total (2 * tr + 1) + 4 * tr + (2 * tr + 1) = 8 * tr + 2. It looks it close to doubling of host memory read traffic mostly compensates speed gain from hardware motion search.

The best solution (in future) - put all internal memory traffic in HWAcc board memory. It will have N frames in -> N frames out. Speed host memory limited close to any AVS filter like 'Levels'. But it need more DirectX texture operating programming. The SAD is simple dot-product of 2 matrixes blocks and alpha-blending of blocks is the natural operation for almost any graphics-accelerator. The only need to found how to select blocks of texture-resource for processing based on x,y coordinates of block inside resource. May be address texture coordinates for block-sized processing with disabled texture filtering. Need to read many new graphics API for possible ways or help of experienced DirectX/Direct3D programmer.

Faster to implement solution - put SAD calculation inside MDegrain memory reading (at least read stripes of blocks to calculate both SAD+weighting + blocks blending from L2/L3 CPU caches). It will limit memory traffic to:
1. DX12_ME read (convert to NV12) for uploading is (2 * tr ) ref + 1 source = (2 * tr + 1)
2. MDegrain (no-overlap) SAD + weight calculation and frames blending - (2 * tr + 1) memory read ops
total - about 4 * tr + 2 - about doubling speed limited by memory read from current testbuild.

But it will break compatibility of MAnalyse-server with other clients of mvtools environment currently (if they use both MVs and SAD from MAnalyse output - not know which may use).

Even more best solution - but in the unknown future: Ask AVS core developers to support AVS memory management of frames in DirectX resources - so a sequencies of filters may be processed in HWAcc memory/board without upload/download to host CPU memory. It is possibly now somehow implemented for CUDA but it is possibly limited to NVIDIA chips only.

[Dogway]

Maaaan, so tired. Finished a first version of HSVxHSV.
I'm not sure it works with current ExTools as I had to make a few changes but I created a video of the tool.
Since this is a concept of the color grading field maybe some of you might not be acquainted with these type of operators. Basically you change Hue, Saturation or Value, depending on a narrow portion of Hue, Saturation or Value. The most common operator that you might see most is HUExSAT but here there are all 9 of them.

modeAxHUE operators are still not included because it requires a bit more work (explicit U and V expressions) and also I'm a bit tired as of current.

I made a video so you get the grasp of it, maybe not the best example as there is not much HUE or SAT variation in the frame.

https://youtu.be/2OO05kFcdVE


@kedautinh12: I think there might be an issue on your end. Try to debug it as I explained above, line by line.

[kedautinh12]

I think if change yuv411 to yuv420 in matchclip. error won't exist anymore
function MatchClip (clip a, clip b, string "kernel", string "matrix", bool "props", bool "length", bool "fast") {

krn = Default (kernel, "Spline16")
fst = Default (fast, true)
fp = Default (props, true)
mfc = Default (length, false) # Match also framecount

YUY2 = b.IsYUY2()
rgbb = b.isRGB() rgba = a.isRGB()
w = b.width() wa = a.width()
h = b.height() ha = a.height()
fc = b.FrameCount() fca = a.FrameCount()
p_type = b.PixelType() p_typea = a.PixelType()
bib = b.BitsPerComponent() bia = a.BitsPerComponent()

propIDs = color_propGet(b)

fs = propNumElements(a,"_ColorRange") > 0 ? \
propGetInt (a,"_ColorRange") == 0 : rgba

fd = propIDs[6]

isUHD = (w > 2599 || h > 1499)
isUHDa = (wa > 2599 || ha > 1499)

mat = Default(matrix, propIDs[2])
mat = ReplaceStr(mat, "Rec", "")


a
yv420 = b.is420() yv420a = a.is420()
yv444 = b.is444() yv444a = a.is444()
!yv420 && bib > bia ? ConvertBits(bib, fulls=fs, fulld=fs) : last

if (!fst) {

ConvertFormat(w, h, fmt_in=p_typea, fmt_out=p_type, cs_in=mat, cs_out="", kernel=krn, tv_range_in=!fs, tv_range_out=!fd, fulls=fs, fulld=fd, show=false)

} else {

rgbp = rgbb && b.isPlanar()
isScl = w != wa || h != ha
mod = rgba && rgbb || isy(a) && isy(b) ? 1 : 2

fmt = Format_fuzzy_search(b, p_type, bib)
fmta = Format_fuzzy_search(a, p_typea, bia)
bc = bicubic_coeffs(krn)
krn = bc[1]>=0. ? "Bicubic" : krn

cplace = rgbb || yv444 || yv420 ? "MPEG1" : isUHD ? "top_left" : "MPEG2"
cplacea = rgba || yv444a || yv420a ? "MPEG1" : isUHDa ? "top_left" : "MPEG2"
not42 = LeftStr(fmta[1],2) != "42"

# Match dimensions
isScl ? w > h ? RatioResize(w,mode="adjust2w",kernel=krn,mod=mod,b=bc[0],c=bc[1]) : \
RatioResize(h,mode="adjust2h",kernel=krn,mod=mod,b=bc[0],c=bc[1]) : last
PadResize(w,h,mode="dilate", mod=mod)

# ConvertBackToYUY2 bug: https://forum.doom9.org/showthread.php?p=1038027#post1038027
isy(b) ? rgba ? DotClip([0.298903,0.586620,0.114477]) : ConvertToY(mat) : \
rgbb ? rgbp ? Eval( "ConvertToPlanarRGB (mat,interlaced=false"+(yv420a || rgba ? ")" : ",ChromaInPlacement=cplacea, chromaresample=krn)")) : \
Eval( "ConvertTo" +string(fmt[3])+"(mat,interlaced=false"+(yv420a || rgba ? ")" : ",ChromaInPlacement=cplacea, chromaresample=krn)")) : \
yv420 ? Eval( "ConvertToYUV420 (false,mat," +(not42 ? "" : """ChromaInPlacement=cplacea, """)+"chromaresample=krn)") : \
YUY2 && rgba ? Eval("""ConvertBackToYUY2 ( mat """) : \
Eval( "ConvertToYUV"+string(fmt[1])+"(false,mat" +(not42 ? "" : ",ChromaInPlacement=cplacea")+(yv444 ? ")" : ",chromaresample=krn, ChromaOutPlacement=cplace)"))
}

clr = !rgbb ? fd ? $008080 : $108080 : \
fd ? $000000 : $101010

!yv420 ? bia > bib ? ConvertBits(bib, dither=1, fulls=fd, fulld=fd) : \
ConvertBits(bib, fulls=fd, fulld=fd) : last
mfc ? fca > fc ? Trim(last, fc-1) : last+BlankClip(b,fc-1,color_yuv=clr) : last

# Planar to Interleaved
YUY2 && !rgba ? ConvertToYUY2() : last

# Frame Properties
if (fst && fp) {
ID = Color_Fuzzy_search(mat)
PropCopy(b)
fd != fs ? PropSet("_ColorRange", fd ? 1 : 0) : last # Overrides the Convertto props
PropSet("_Matrix", ID[1]) # Overrides the Convertto props
PropSet("_Primaries", ID[2]) } } # Disable this line and above when convertto supports writing to RGB frameprops (currently RGB is hardcoded to _Matrix 0 and _Primaries 2 -undefined-)

[anton_foy]

Maaaan, so tired. Finished a first version of HSVxHSV.
I'm not sure it works with current ExTools as I had to make a few changes but I created a video of the tool.
Since this is a concept of the color grading field maybe some of you might not be acquainted with these type of operators. Basically you change Hue, Saturation or Value, depending on a narrow portion of Hue, Saturation or Value. The most common operator that you might see most is HUExSAT but here there are all 9 of them.

modeAxHUE operators are still not included because it requires a bit more work (explicit U and V expressions) and also I'm a bit tired as of current.

I made a video so you get the grasp of it, maybe not the best example as there is not much HUE or SAT variation in the frame.

https://youtu.be/2OO05kFcdVE


Wow great work! Wonder how this will result when denoising in this mode.
I will put to the test in the evening.

[Dogway]

@anton_foy: It's to colors what ex_luts() is to convolutions, a powerhouse filter. With show=2 you can reuse the mask for further filtering also. I will try today to finish the modeAxHUE and redo the video with a better sample with more variance on HUE and SAT. I will also reparametrize 'dA' ( I will rename to degrees or so ) because HUE and SAT values mostly live in the greys so like 1/3 of the dA range is unutilized. When I finish I will post some images here showing the latest filters.

@kedautinh12: If that works for you, fine. But it's wrong, you are going to have further issues with that, to name one YUV420 doesn't use MPEG1 chroma location. Aside from that you won't be able to MatchClip() to a 411 format.

[StainlessS]

I think if change yuv411 to yuv420 in matchclip. error won't exist anymore
Maybe you should hi-lite the place where you changed YUV411 to YUV420.
[EDIT: There is no instance of "411" in your posted script]

EDIT: And YV411 should only really be used for input, and converted to something else [a horrible colourspace].

[kedautinh12]

i know what affect to matchclip. Thanks

[Dogway]

After 5 restless days I finished HSVxHSV. Now all operators work over any bitdepth. For matching operators it's about double the speed of Tweak(). I made a new video showing the different operators. It's a bit random since it isn't supposed to be a tutorial, but you can see in the color plots how the sliders behave and how is my workflow by checking the mask back and forth.
http://i.imgur.com/b6ayieGm.png (https://youtu.be/nMWTjdoKUBU)

To finish it I had to improve a few other functions for ExTools which in turn benefits overall all other packs, so TransformsPack was also updated.

I also want to showcase GamutWarning(), I implemented the EBU R-103 normative as well which is more lenient on limits.
http://i.imgur.com/ke8nVCUm.png (https://i.imgur.com/ke8nVCU.png)
And with EBU R-103
http://i.imgur.com/wp0jo2tm.png (https://i.imgur.com/wp0jo2t.png)


Finally I added x and y axis to the Curve Fitting Evaluator and fixed the 'positive' argument so it can also be scaled. This will come useful when debugging HDR.
https://i.imgur.com/lORJD55l.png

Here is ex_contrast() showing the curve plot by using Curve Fitting Evaluator. It's integrated so simply call ex_contrast(0.7,show=true) (notice how the curve is within tv range [16-235] since the input is tv range as well)
https://i.imgur.com/71mVgh3l.png

[Dogway]

@wonkey_monkey: I made a custom 48 inputs ranking order sorting network, I thought you might be interested to include it in the web app. It was a hard task because it uses two 24 inputs sorting networks but then you have to merge them, for this a Batcher Odd-Even sorting network should suffice but all the tools I found online only work with powers of 2, so 32 then 64, but 48 wasn't working. My idea was to add a before-last step of comparing inputs with a stride of 3, before final comparison of stride 2.

The only part I didn't do was multi-threading optimization or "layering" (interleaving the comparisons). I didn't find out by what mechanism it optimizes multi-threading so maybe you can find a pattern with your tool, specially those that are merged with a batcher odd-even merge method. After this if you are interested you can derive 47, 46, 45 or 44 inputs SN by deleting one last variable.

The other thing is this list skips variables X, Y and XX due to constraints with my filters, you might want to add those back.

48 inputs sorting network:
D Z dup1 dup1 min D^ max Z^
L W dup1 dup1 min L^ max W^
H V dup1 dup1 min H^ max V^
A U dup1 dup1 min A^ max U^
R T dup1 dup1 min R^ max T^
N S dup1 dup1 min N^ max S^
C Q dup1 dup1 min C^ max Q^
J P dup1 dup1 min J^ max P^
I O dup1 dup1 min I^ max O^
B M dup1 dup1 min B^ max M^
F K dup1 dup1 min F^ max K^
E G dup1 dup1 min E^ max G^
U Z dup1 dup1 min U^ max Z^
M W dup1 dup1 min M^ max W^
Q V dup1 dup1 min Q^ max V^
G T dup1 dup1 min G^ max T^
K S dup1 dup1 min K^ max S^
E R dup1 dup1 min E^ max R^
O P dup1 dup1 min O^ max P^
F N dup1 dup1 min F^ max N^
B L dup1 dup1 min B^ max L^
I J dup1 dup1 min I^ max J^
C H dup1 dup1 min C^ max H^
A D dup1 dup1 min A^ max D^
W Z dup1 dup1 min W^ max Z^
T V dup1 dup1 min T^ max V^
L U dup1 dup1 min L^ max U^
P S dup1 dup1 min P^ max S^
O R dup1 dup1 min O^ max R^
N Q dup1 dup1 min N^ max Q^
D M dup1 dup1 min D^ max M^
H K dup1 dup1 min H^ max K^
G J dup1 dup1 min G^ max J^
F I dup1 dup1 min F^ max I^
C E dup1 dup1 min C^ max E^
A B dup1 dup1 min A^ max B^
S V dup1 dup1 min S^ max V^
P T dup1 dup1 min P^ max T^
M R dup1 dup1 min M^ max R^
J Q dup1 dup1 min J^ max Q^
H O dup1 dup1 min H^ max O^
G L dup1 dup1 min G^ max L^
E I dup1 dup1 min E^ max I^
C F dup1 dup1 min C^ max F^
P W dup1 dup1 min P^ max W^
J U dup1 dup1 min J^ max U^
Q T dup1 dup1 min Q^ max T^
D O dup1 dup1 min D^ max O^
L N dup1 dup1 min L^ max N^
K M dup1 dup1 min K^ max M^
B I dup1 dup1 min B^ max I^
E H dup1 dup1 min E^ max H^
Q Z dup1 dup1 min Q^ max Z^
S W dup1 dup1 min S^ max W^
T U dup1 dup1 min T^ max U^
M R dup1 dup1 min M^ max R^
I P dup1 dup1 min I^ max P^
J O dup1 dup1 min J^ max O^
K N dup1 dup1 min K^ max N^
G L dup1 dup1 min G^ max L^
A H dup1 dup1 min A^ max H^
B F dup1 dup1 min B^ max F^
D E dup1 dup1 min D^ max E^
V Z dup1 dup1 min V^ max Z^
R W dup1 dup1 min R^ max W^
Q T dup1 dup1 min Q^ max T^
O S dup1 dup1 min O^ max S^
N P dup1 dup1 min N^ max P^
I K dup1 dup1 min I^ max K^
F J dup1 dup1 min F^ max J^
E H dup1 dup1 min E^ max H^
B G dup1 dup1 min B^ max G^
A C dup1 dup1 min A^ max C^
U V dup1 dup1 min U^ max V^
S T dup1 dup1 min S^ max T^
P R dup1 dup1 min P^ max R^
O Q dup1 dup1 min O^ max Q^
M N dup1 dup1 min M^ max N^
K L dup1 dup1 min K^ max L^
H J dup1 dup1 min H^ max J^
G I dup1 dup1 min G^ max I^
E F dup1 dup1 min E^ max F^
C D dup1 dup1 min C^ max D^
V W dup1 dup1 min V^ max W^
R U dup1 dup1 min R^ max U^
N T dup1 dup1 min N^ max T^
P Q dup1 dup1 min P^ max Q^
M O dup1 dup1 min M^ max O^
J L dup1 dup1 min J^ max L^
E K dup1 dup1 min E^ max K^
H I dup1 dup1 min H^ max I^
D G dup1 dup1 min D^ max G^
B C dup1 dup1 min B^ max C^
U V dup1 dup1 min U^ max V^
N S dup1 dup1 min N^ max S^
Q R dup1 dup1 min Q^ max R^
O P dup1 dup1 min O^ max P^
F K dup1 dup1 min F^ max K^
I J dup1 dup1 min I^ max J^
G H dup1 dup1 min G^ max H^
C D dup1 dup1 min C^ max D^
T U dup1 dup1 min T^ max U^
Q S dup1 dup1 min Q^ max S^
L N dup1 dup1 min L^ max N^
K M dup1 dup1 min K^ max M^
F H dup1 dup1 min F^ max H^
D E dup1 dup1 min D^ max E^
R T dup1 dup1 min R^ max T^
N P dup1 dup1 min N^ max P^
L O dup1 dup1 min L^ max O^
J M dup1 dup1 min J^ max M^
I K dup1 dup1 min I^ max K^
E G dup1 dup1 min E^ max G^
R S dup1 dup1 min R^ max S^
P Q dup1 dup1 min P^ max Q^
N O dup1 dup1 min N^ max O^
L M dup1 dup1 min L^ max M^
J K dup1 dup1 min J^ max K^
H I dup1 dup1 min H^ max I^
F G dup1 dup1 min F^ max G^

UU AA dup1 dup1 min UU^ max AA^
MM BB dup1 dup1 min MM^ max BB^
QQ CC dup1 dup1 min QQ^ max CC^
YY DD dup1 dup1 min YY^ max DD^
GG EE dup1 dup1 min GG^ max EE^
KK FF dup1 dup1 min KK^ max FF^
VV HH dup1 dup1 min VV^ max HH^
OO II dup1 dup1 min OO^ max II^
PP JJ dup1 dup1 min PP^ max JJ^
WW LL dup1 dup1 min WW^ max LL^
SS NN dup1 dup1 min SS^ max NN^
TT RR dup1 dup1 min TT^ max RR^
DD AA dup1 dup1 min DD^ max AA^
LL BB dup1 dup1 min LL^ max BB^
HH CC dup1 dup1 min HH^ max CC^
RR EE dup1 dup1 min RR^ max EE^
NN FF dup1 dup1 min NN^ max FF^
TT GG dup1 dup1 min TT^ max GG^
JJ II dup1 dup1 min JJ^ max II^
SS KK dup1 dup1 min SS^ max KK^
WW MM dup1 dup1 min WW^ max MM^
PP OO dup1 dup1 min PP^ max OO^
VV QQ dup1 dup1 min VV^ max QQ^
YY UU dup1 dup1 min YY^ max UU^
BB AA dup1 dup1 min BB^ max AA^
EE CC dup1 dup1 min EE^ max CC^
MM DD dup1 dup1 min MM^ max DD^
II FF dup1 dup1 min II^ max FF^
JJ GG dup1 dup1 min JJ^ max GG^
KK HH dup1 dup1 min KK^ max HH^
UU LL dup1 dup1 min UU^ max LL^
QQ NN dup1 dup1 min QQ^ max NN^
RR OO dup1 dup1 min RR^ max OO^
SS PP dup1 dup1 min SS^ max PP^
VV TT dup1 dup1 min VV^ max TT^
YY WW dup1 dup1 min YY^ max WW^
FF CC dup1 dup1 min FF^ max CC^
II EE dup1 dup1 min II^ max EE^
LL GG dup1 dup1 min LL^ max GG^
OO HH dup1 dup1 min OO^ max HH^
QQ JJ dup1 dup1 min QQ^ max JJ^
RR MM dup1 dup1 min RR^ max MM^
TT PP dup1 dup1 min TT^ max PP^
VV SS dup1 dup1 min VV^ max SS^
II BB dup1 dup1 min II^ max BB^
OO DD dup1 dup1 min OO^ max DD^
HH EE dup1 dup1 min HH^ max EE^
UU JJ dup1 dup1 min UU^ max JJ^
MM KK dup1 dup1 min MM^ max KK^
NN LL dup1 dup1 min NN^ max LL^
WW PP dup1 dup1 min WW^ max PP^
TT QQ dup1 dup1 min TT^ max QQ^
HH AA dup1 dup1 min HH^ max AA^
FF BB dup1 dup1 min FF^ max BB^
EE DD dup1 dup1 min EE^ max DD^
LL GG dup1 dup1 min LL^ max GG^
PP II dup1 dup1 min PP^ max II^
OO JJ dup1 dup1 min OO^ max JJ^
NN KK dup1 dup1 min NN^ max KK^
RR MM dup1 dup1 min RR^ max MM^
YY QQ dup1 dup1 min YY^ max QQ^
WW SS dup1 dup1 min WW^ max SS^
UU TT dup1 dup1 min UU^ max TT^
CC AA dup1 dup1 min CC^ max AA^
GG BB dup1 dup1 min GG^ max BB^
HH EE dup1 dup1 min HH^ max EE^
JJ FF dup1 dup1 min JJ^ max FF^
KK II dup1 dup1 min KK^ max II^
PP NN dup1 dup1 min PP^ max NN^
SS OO dup1 dup1 min SS^ max OO^
TT QQ dup1 dup1 min TT^ max QQ^
WW RR dup1 dup1 min WW^ max RR^
YY VV dup1 dup1 min YY^ max VV^
DD CC dup1 dup1 min DD^ max CC^
FF EE dup1 dup1 min FF^ max EE^
II GG dup1 dup1 min II^ max GG^
JJ HH dup1 dup1 min JJ^ max HH^
LL KK dup1 dup1 min LL^ max KK^
NN MM dup1 dup1 min NN^ max MM^
QQ OO dup1 dup1 min QQ^ max OO^
RR PP dup1 dup1 min RR^ max PP^
TT SS dup1 dup1 min TT^ max SS^
VV UU dup1 dup1 min VV^ max UU^
CC BB dup1 dup1 min CC^ max BB^
GG DD dup1 dup1 min GG^ max DD^
KK EE dup1 dup1 min KK^ max EE^
II HH dup1 dup1 min II^ max HH^
LL JJ dup1 dup1 min LL^ max JJ^
OO MM dup1 dup1 min OO^ max MM^
TT NN dup1 dup1 min TT^ max NN^
QQ PP dup1 dup1 min QQ^ max PP^
UU RR dup1 dup1 min UU^ max RR^
WW VV dup1 dup1 min WW^ max VV^
DD CC dup1 dup1 min DD^ max CC^
KK FF dup1 dup1 min KK^ max FF^
HH GG dup1 dup1 min HH^ max GG^
JJ II dup1 dup1 min JJ^ max II^
SS NN dup1 dup1 min SS^ max NN^
PP OO dup1 dup1 min PP^ max OO^
RR QQ dup1 dup1 min RR^ max QQ^
VV UU dup1 dup1 min VV^ max UU^
EE DD dup1 dup1 min EE^ max DD^
HH FF dup1 dup1 min HH^ max FF^
MM KK dup1 dup1 min MM^ max KK^
NN LL dup1 dup1 min NN^ max LL^
SS QQ dup1 dup1 min SS^ max QQ^
UU TT dup1 dup1 min UU^ max TT^
GG EE dup1 dup1 min GG^ max EE^
KK II dup1 dup1 min KK^ max II^
MM JJ dup1 dup1 min MM^ max JJ^
OO LL dup1 dup1 min OO^ max LL^
PP NN dup1 dup1 min PP^ max NN^
TT RR dup1 dup1 min TT^ max RR^
GG FF dup1 dup1 min GG^ max FF^
II HH dup1 dup1 min II^ max HH^
KK JJ dup1 dup1 min KK^ max JJ^
MM LL dup1 dup1 min MM^ max LL^
OO NN dup1 dup1 min OO^ max NN^
QQ PP dup1 dup1 min QQ^ max PP^
SS RR dup1 dup1 min SS^ max RR^



A B dup1 dup1 min A^ max B^
C D dup1 dup1 min C^ max D^
E F dup1 dup1 min E^ max F^
G H dup1 dup1 min G^ max H^
I J dup1 dup1 min I^ max J^
K L dup1 dup1 min K^ max L^
M N dup1 dup1 min M^ max N^
O P dup1 dup1 min O^ max P^
Q R dup1 dup1 min Q^ max R^
S T dup1 dup1 min S^ max T^
U V dup1 dup1 min U^ max V^
W Z dup1 dup1 min W^ max Z^
AA BB dup1 dup1 min AA^ max BB^
CC DD dup1 dup1 min CC^ max DD^
EE FF dup1 dup1 min EE^ max FF^
GG HH dup1 dup1 min GG^ max HH^
II JJ dup1 dup1 min II^ max JJ^
KK LL dup1 dup1 min KK^ max LL^
MM NN dup1 dup1 min MM^ max NN^
OO PP dup1 dup1 min OO^ max PP^
QQ RR dup1 dup1 min QQ^ max RR^
SS TT dup1 dup1 min SS^ max TT^
UU VV dup1 dup1 min UU^ max VV^
WW YY dup1 dup1 min WW^ max YY^
B II dup1 dup1 min B^ max II^
D KK dup1 dup1 min D^ max KK^
F MM dup1 dup1 min F^ max MM^
H OO dup1 dup1 min H^ max OO^
J QQ dup1 dup1 min J^ max QQ^
L SS dup1 dup1 min L^ max SS^
N UU dup1 dup1 min N^ max UU^
P WW dup1 dup1 min P^ max WW^
B Q dup1 dup1 min B^ max Q^
D S dup1 dup1 min D^ max S^
F U dup1 dup1 min F^ max U^
H W dup1 dup1 min H^ max W^
J AA dup1 dup1 min J^ max AA^
L CC dup1 dup1 min L^ max CC^
N EE dup1 dup1 min N^ max EE^
P GG dup1 dup1 min P^ max GG^
R II dup1 dup1 min R^ max II^
T KK dup1 dup1 min T^ max KK^
V MM dup1 dup1 min V^ max MM^
Z OO dup1 dup1 min Z^ max OO^
BB QQ dup1 dup1 min BB^ max QQ^
DD SS dup1 dup1 min DD^ max SS^
FF UU dup1 dup1 min FF^ max UU^
HH WW dup1 dup1 min HH^ max WW^
B I dup1 dup1 min B^ max I^
D K dup1 dup1 min D^ max K^
F M dup1 dup1 min F^ max M^
H O dup1 dup1 min H^ max O^
J Q dup1 dup1 min J^ max Q^
L S dup1 dup1 min L^ max S^
N U dup1 dup1 min N^ max U^
P W dup1 dup1 min P^ max W^
R AA dup1 dup1 min R^ max AA^
T CC dup1 dup1 min T^ max CC^
V EE dup1 dup1 min V^ max EE^
Z GG dup1 dup1 min Z^ max GG^
BB II dup1 dup1 min BB^ max II^
DD KK dup1 dup1 min DD^ max KK^
FF MM dup1 dup1 min FF^ max MM^
HH OO dup1 dup1 min HH^ max OO^
JJ QQ dup1 dup1 min JJ^ max QQ^
LL SS dup1 dup1 min LL^ max SS^
NN UU dup1 dup1 min NN^ max UU^
PP WW dup1 dup1 min PP^ max WW^
B E dup1 dup1 min B^ max E^
D G dup1 dup1 min D^ max G^
F I dup1 dup1 min F^ max I^
H K dup1 dup1 min H^ max K^
J M dup1 dup1 min J^ max M^
L O dup1 dup1 min L^ max O^
N Q dup1 dup1 min N^ max Q^
P S dup1 dup1 min P^ max S^
R U dup1 dup1 min R^ max U^
T W dup1 dup1 min T^ max W^
V AA dup1 dup1 min V^ max AA^
Z CC dup1 dup1 min Z^ max CC^
BB EE dup1 dup1 min BB^ max EE^
DD GG dup1 dup1 min DD^ max GG^
FF II dup1 dup1 min FF^ max II^
HH KK dup1 dup1 min HH^ max KK^
JJ MM dup1 dup1 min JJ^ max MM^
LL OO dup1 dup1 min LL^ max OO^
NN QQ dup1 dup1 min NN^ max QQ^
PP SS dup1 dup1 min PP^ max SS^
RR UU dup1 dup1 min RR^ max UU^
TT WW dup1 dup1 min TT^ max WW^
B C dup1 dup1 min B^ max C^
D E dup1 dup1 min D^ max E^
F G dup1 dup1 min F^ max G^
H I dup1 dup1 min H^ max I^
J K dup1 dup1 min J^ max K^
L M dup1 dup1 min L^ max M^
N O dup1 dup1 min N^ max O^
P Q dup1 dup1 min P^ max Q^
R S dup1 dup1 min R^ max S^
T U dup1 dup1 min T^ max U^
V W dup1 dup1 min V^ max W^
Z AA dup1 dup1 min Z^ max AA^
BB CC dup1 dup1 min BB^ max CC^
DD EE dup1 dup1 min DD^ max EE^
FF GG dup1 dup1 min FF^ max GG^
HH II dup1 dup1 min HH^ max II^
JJ KK dup1 dup1 min JJ^ max KK^
LL MM dup1 dup1 min LL^ max MM^
NN OO dup1 dup1 min NN^ max OO^
PP QQ dup1 dup1 min PP^ max QQ^
RR SS dup1 dup1 min RR^ max SS^
TT UU dup1 dup1 min TT^ max UU^
VV WW dup1 dup1 min VV^ max WW^

[Blankmedia]

I might be crazy, but I think the new ex_tools broke smdegrain. Forget the error code I'll check tomorrow

[Dogway]

Break, I don't know but I have to rebase for prefilter=0 and prefilter=4, that's about what I found, did you find something else?


EDIT: I updated SMDegrain with default overlap=blksize/4 when refinemotion is used. There's a slight increase in detail retention (high and low freq) when there's motion, also a bit less denoising but minimum, so worth it IMO. I also added the curve plot to ex_luma_rebuild() when used as a standalone and also with SMDegrain(show=true), so you know what's going on underneath. Also fixed the small issues related to the 'fulls' args.

[tormento]

I updated SMDegrain
Updated all your scripts to latest versions.

With

SMDegrain (tr=6, thSAD=600, refinemotion=true, contrasharp=false, PreFilter=6, plane=4, chroma=true, LFR=true, DCTFlicker=true)

I get:

Script Error

I don't know what 'fs' means.

(D:/Programmi/Media/AviSynth+/plugins64/ExTools-7.2~Dogway.avsi, line 5678)
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.4.1d~Dogway.avsi, line 411)
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.4.1d~Dogway.avsi, line 416)
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.4.1d~Dogway.avsi, line 416)
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.4.1d~Dogway.avsi, line 421)
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.4.1d~Dogway.avsi, line 612)

It doesn't happen for:

SMDegrain (tr=6, thSAD=600, refinemotion=true, contrasharp=false, PreFilter=6, plane=4, chroma=true)

[Dogway]

Yep, thank you! it's a typo on line 5678 of ExTools, should be " str = ex_dlut (str, bi, un)"

[tormento]

Yep, thank you! it's a typo on line 5678 of ExTools, should be " str = ex_dlut (str, bi, un)"
Thanks, I will fix by hand waiting for yours.

[Blankmedia]

Thank you for all your nice work, do you think a non binarized mt_hysteresis would be possible. Like we could set threshold for each layer, it would then return Us with the original softer mask.

Most of all would this be pertinent.

[Dogway]

At the time I had a look at mt_hysteresis source code and didn't understand a thing, well maybe only that uses lots of loops so this is not something that can be translated well to AVS syntax, at least efficiently. This is something for plugin developers.
Non-binarized hysteresis sounds strange, what's your use case?

By the way, I released PseudoColor() to map luminosity to a rainbow type heat map, useful for luminosity segmentation or to augment luma ranges and devise its isoparams, but specially useful to debug algorithms such as blend modes (https://trac.ffmpeg.org/wiki/Blend) or gradient magnitudes (http://waynegm.github.io/OpendTect-Plugin-Docs/plugins/GradientAttrib.html), etc. It's fast because it doesn't go through RGB.

http://i.imgur.com/QJYtMuvm.png (https://i.imgur.com/QJYtMuv.png)

EDIT: Also I'm going to start running zopti for optimal frame interpolation, I have a few ideas, not that it will replace FRC but I will experiment a bit.

[Boulder]

Is the "change limiter" in SPresso somehow possible to turn into a helper function? I did try real.finders sh_limitchange, but it's not working as well as the limit of change in SPresso (somehow starts working only at limit 0.5).

[Dogway]

sh_LimitChange looks to me like simple ex_clamp/mt_clamp. In SPresso there are two modes with negative and positive 'Limit' and also additional 'Bias' args. To be honest I haven't researched the algo I simply optimized from Didee's original. There's also a limiter by cretindesalpes that I haven't ported.

Check here (https://github.com/Dogway/Avisynth-Scripts/blob/master/MIX%20mods/Spresso.avsi) if it works fine for you.

[Boulder]

sh_LimitChange looks to me like simple ex_clamp/mt_clamp. In SPresso there are two modes with negative and positive 'Limit' and also additional 'Bias' args. To be honest I haven't researched the algo I simply optimized from Didee's original. There's also a limiter by cretindesalpes that I haven't ported.

Check here (https://github.com/Dogway/Avisynth-Scripts/blob/master/MIX%20mods/Spresso.avsi) if it works fine for you.

Thank you, seems to work fine. I can now implement this part in my denoiser much better - i.e. allow the function to utilize the mix of blur+sharpen idea from MCDegrainSharp at full strength and limit the result of the whole process. MDegrains internal limit makes sharpening the super clip pretty much unusable if the limit is low (which it always is in my case).

[real.finder]

sh_LimitChange looks to me like simple ex_clamp/mt_clamp. In SPresso there are two modes with negative and positive 'Limit' and also additional 'Bias' args. To be honest I haven't researched the algo I simply optimized from Didee's original. There's also a limiter by cretindesalpes that I haven't ported.

Check here (https://github.com/Dogway/Avisynth-Scripts/blob/master/MIX%20mods/Spresso.avsi) if it works fine for you.

I think ex_limitchange should be in ExTools

as for "limiter by cretindesalpes" https://github.com/realfinder/AVS-Stuff/blob/832a3e0d9ff501c04483cb05a34055d121ce531e/avs%202.5%20and%20up/Zs_RF_Shared.avsi#L476

[Dogway]

Thanks! I'm a bit busy with Zopti currently but I will try to optimize when I can. Tuning MAnalyse is important as it impacts QTGMC, FrameRateConverter and SMDegrain by picking the optimal pareto of the convex hull. I also want to finish the Harris corner detection and SAD Analysis tool.
I normally don't add ports to ExTools or at least expressions that I don't have full understanding of, I will check what I do as a limiter is some kind of a mask so also considered MasksPack.

[anton_foy]

Your mod of ChromaReconstructor_faster() is not available anymore? It looks like it does not have a function in the latest transformspack.avsi.

[Dogway]

Thanks for heads up. I fixed "ChromaReconstructor v3mod" and also a few things in TransformsPack. Should work fine now, let me know.

[Boulder]

Is there something broken with ex_minblur, or have I copy-pasted/updated something incorrectly? In this example, the clip 'prefilt' is very much washed out, and if you comment ex_minblur out, it's ok.

clp=DGSource("test.dgi") # regular 8-bit source
prefiltbits=8
orgbits=8

prefilt = clp.convertbits(bits=prefiltbits)
prefilt = prefilt.ex_minblur(r=2, uv=3)
prefilt = prefilt.ex_luma_rebuild(s0=3.0, uv=3)
prefilt = prefilt.convertbits(orgbits)
clp = clp.ex_luma_rebuild(s0=3.0, uv=3)

interleave(clp, prefilt)

[Dogway]

Loader probably didn't write "_ColorRange" frameprops, then ex_minblur() wrongly assumes PC level and writes the frameprops (will investigate this), and the brightening is performed by ex_luma_rebuild() reading the wrong frameprops, for the time being use ex_luma_rebuild(s0=3.0, uv=3, tv_range=true)

EDIT: The output of ex_minblur() is inheriting the frameprops of the masks (ex_makediff()) so that's why it's "_ColorRange" = 0. By changing the clip order this is fixed. Normally I pay attention to this but it might happen somewhere else.
Fixed line in ex_minblur():
ex_lutxyz(clp,rg12D,RG4D,"y "+bi32+" A@ x z - B@ * 0 < x A abs B abs < x A - z ? ?",Y=y,UV=uv,fulls=fs)

[Boulder]

Loader probably didn't write "_ColorRange" frameprops, then ex_minblur() wrongly assumes PC level and writes the frameprops (will investigate this), and the brightening is performed by ex_luma_rebuild() reading the wrong frameprops, for the time being use ex_luma_rebuild(s0=3.0, uv=3, tv_range=true)

PropShow does indicate _ColorRange=1 after loading the source. Using it after ex_minblur shows that the function overwrites the information with _ColorRange=0, so there's something strange happening there.

[Dogway]

Check now, I updated SMDegrain(). Results from Zopti will come on a later version fro v3.5.0.

[Boulder]

Check now, I updated SMDegrain(). Results from Zopti will come on a later version fro v3.5.0.

Thanks, it's working now. Looking forward to seeing what you find out with Zopti :)

[anton_foy]

Thanks for heads up. I fixed "ChromaReconstructor v3mod" and also a few things in TransformsPack. Should work fine now, let me know.

Thanks but I get an error Script error: there is no function named 'SI_physicalcores' on line 105 of the chromareconstructor_v3.2.avsi
My Avs+ version is this Avisynth 3.7.1 FINAL x64 (https://github.com/AviSynth/AviSynthPlus/releases/tag/v3.7.1)

[kedautinh12]

I think it's belong to Sysinfo
https://forum.doom9.org/showthread.php?t=176131

[anton_foy]

I think it's belong to Sysinfo
https://forum.doom9.org/showthread.php?t=176131

Thanks! Now this: KNLMEANSCL:'rclip' doesn't match the source clip! line 120.
Maybe because my clip is 3840x2160?

[Dogway]

What's your call and clip format? is it YV12?

EDIT: Fixed it. Check here (https://github.com/Dogway/Avisynth-Scripts/blob/master/MIX%20mods/ChromaReconstructor%20v3mod.avsi).

[mp3dom]

Uhm, maybe I'm missing something, but just a plain:

colorbars(width=1920,height=1080,pixel_type="yv16")
qtgmc()

gives me a

System Exception - Access Violation
ExTools.avsi at line 67
SMDegrain.avsi at line 797
QTGMC.avsi at line 643

Plugins/scripts are all at the latest versions and AVS+ is 3.7.1

[Dogway]

Works fine here. The error points to ex_luma_rebuild() and this in turn to ex_lut(). Try with a simple call like: "ex_Luma_Rebuild(S0=3.0, uv=3, tv_range=true)"

[mp3dom]

I get the exact same error (at the same lines), but without the last error that refers to QTGMC.avsi)

[Dogway]

If you are on AVS+ v3.7.1 final, SMDegrain v3.4.3 and ExTools v7.2 I might try to test with AVSmeter to check it's not a conflict with AvspMod or the editor you use. You are using AVS+ x64 right?

Try this:
s0= 3.0
c = 1.0/16
k = (s0 - 1) * c
l = 1 - k
d = 1 + c
f = d * c
Expr(Format("f32 {k} {d} {f} x ymin - 0 max ymax ymin - / A@ {c} + / - * A {l} * + range_max *"), "x cmin - range_max cmax cmin - / *", \
scale_inputs="none", clamp_float=false, lut=1)

[mp3dom]

Yeah, AVS is the final 3.7.1, SMDegrain is 3.4.3d, ExTools 7.2 and QTGMC 3.73. Had same errors on both x86/x64 with VDub2 but something strange happened. I've reinstalled 3.7.1 final using the installer (previously I replaced just the .dll) and now it works on x64 but still I have the same error under x86, but considering you asked me about x64, I guess that's to be expected?
Thanks!

Edit: Oh, and by the way...
Try this:

This works on x64 but gave me access violation on x86 on the last line (scale_inputs=...)

[Dogway]

anton_foy was having issues before with x86. I think there's something odd in x86 AVS+ but pinterf has to confirm, simply by testing the above Expr() call.


Edit: Oh, and by the way...

This works on x64 but gave me access violation on x86 on the last line (scale_inputs=...)

Ok, so 'scale_inputs' is broken on x86, I will try to ask pinterf when he's around.

[mp3dom]

Oh well, at least now I can use it on x64, which is better than nothing.
Thanks.

[anton_foy]

Thanks for heads up. I fixed "ChromaReconstructor v3mod" and also a few things in TransformsPack. Should work fine now, let me know.

Thanks although I get now: KNLMEANSCL: Fatal error! (AvisynthGetFrame)

EDIT: aha when I use ex_knlmeanscl() I also get this error but using ordinary knlmeanscl() it works but I don't know if that helps.

[Dogway]

Post your call and source resolution.

[anton_foy]

Post your call and source resolution.

Source resolution: 3840x2160

Sorry I decieved you (not on purpose) it was ex_KNLMeansCL(D=2, A=1, h=1, s=6, device_type="gpu") that is giving me the error.

So the following does NOT give me error:
ex_KNLMeansCL()
KNLMeansCL()
KNLMeansCL(D=2, A=1, h=1, s=6, device_type="gpu")

ClearAutoloadDirs()
LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64\mp_pipeline.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64\lsmashsource.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64\gradationcurve_x64.dll")
#LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64\manyPlus.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64\awarpsharpmt.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64\hqdn3d.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64\fft3dgpu.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64\rgtools.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64\masktools2.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64\SysInfo64.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64\nnedi3.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64\deblock-x64.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64\KNLMeansCL.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64\mvtools2.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64\vscube.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64\vscnr2.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64\neo-f3kdb.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64\neo-dfttest.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64\neo-fft3d.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64\reduceflicker.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64\fastblur.dll")
LoadPlugin("C:\Program Files (x86)\AviSynth+\plugins64\dctfilter.dll")
import("C:\Program Files (x86)\AviSynth+\plugins\Deblock_QED_MT2.avsi")
import("C:\Program Files (x86)\AviSynth+\plugins\gradepack.avsi")
import("C:\Program Files (x86)\AviSynth+\plugins\deblockpack.avsi")
import("C:\Program Files (x86)\AviSynth+\plugins\maskspack.avsi")
import("C:\Program Files (x86)\AviSynth+\plugins\extools.avsi")
import("C:\Program Files (x86)\AviSynth+\plugins\knlmc.avsi")
import("C:\Program Files (x86)\AviSynth+\plugins\transformspack.avsi")
import("C:\Program Files (x86)\AviSynth+\plugins\resizerspack.avsi")
import("C:\Program Files (x86)\AviSynth+\plugins\smdegrain.avsi")
import("C:\Program Files (x86)\AviSynth+\plugins\chromareconstrucor.avsi")
import("C:\Program Files (x86)\AviSynth+\plugins\basiq.avsi")


#Index
LSMASHVideoSource("C:\Users\comp\Videos\C0561.mp4")


convertbits(16) #bits=16,fulls=true
Deblock_QED_mt(quant1=26, aOff1=1, aOff2=1, bOff1=2, bOff2=2, uv=3)
Levels(0, 1, 255*256, 0, 235*256, coring=false, dither=false)

#### Denoise ###
#ChromaReconstructor_faster()
ex_KNLMeansCL(D=2, A=1, h=1, s=6, device_type="gpu")
#KNLMeansCL(D=2, A=1, h=1, s=6, device_type="gpu")

#Chroma Upscaling
ConverttoYUV444()
ConvertBits(bits=10, dither=1)

Prefetch (SI_PhysicalCores ())

[Dogway]

Works ok for me with your script and using also a 4K source clip.
Maybe this might be a memory issue, try to play with SetMemoryMax(16384/n) # 'n': 1, 2, 4, 8

In any case the issue might be instead VRam, try to remove all calls except ex_KNLMeansCL(), if that doesn't work, resize your clip to half size and test again if you still get the issue.

[DTL]

New idea about degraining processing with MDegrainN added - https://forum.doom9.org/showthread.php?p=1962418#post1962418 . Also new params to adjust by content. Now it is possible to lower thSAD 'old/general' param significantly to try to keep low contrast textures better while keeping the same or better 'denoise strength' in terms of lower output MPEG encoded speed/filesize. Though still only implemented with MDegrainN (not with fixed-tr MDegrainX). Also speed will be somehow lower because of new conditions checks added.

[tormento]

@Dogway

Am I wrong or did you plan to convert to XYZ color space before denoising?

[Dogway]

XYZ is not a perceptually uniform space, IPT or OkLab are but conversion is slow so I don't know to what degree it might be useful. What I might add is a HUE plane for MAnalyse for better detection of chroma MV, but this has implications of current settings so I need more testing for correlation and magnitudes.

I just uploaded some updates for my other scripts so maybe now I have more time for MVTools tests.

[Boulder]

Can you tell me if the "grid" which appears with ex_makediff is normal with this script? I noticed that it appears with HBD input to SPresso even with rather light options like limit=0.5, I've increased the limit here to make it more obvious.


DGSource("clip.dgi",ct=140,cb=140)
ConvertBits(bits=16)
test=spresso(bias=20, biasc=40, limit=2.5, limitc=5, rgmode=4, rgmodec=4)
ex_makediff(last, test, aug=100)


https://i.ibb.co/ZSPBFKg/lotr-spresso-grid.png (https://ibb.co/ZSPBFKg)


EDIT: hmm.. I suppose they are macroblock boundaries so should be ok.

[Dogway]

Sorry I was watching Nadal's match :rolleyes:
Yes it's normal, they are compression DCT blocks, normally they are not seen but at x100 they are visible. I've seen worse than that example so it's ok.
Here's an HBD vs 8-bit example. Only at x100 you start to see some difference (basically quantization difference) so they are a practical match
test1=spresso(bias=20, biasc=40, limit=2.5, limitc=5, rgmode=4, rgmodec=4).ConvertBits(bits=16)
ConvertBits(bits=16)
test2=spresso(bias=20, biasc=40, limit=2.5, limitc=5, rgmode=4, rgmodec=4)

ex_makediff(test1, test2, metric="none", aug=100, dif=true, show=2)


Some summary of what I'm doing right now:

I'm yet to update a few scripts to level them out to current ExTools and LSFmod, namely SMDegrain and TransformsPack. SharpenersPack has also seen some optimization edits yet to upload.

I updated LSFmod to v5.0, following old Didée's notes and porting some original LSF concepts, mainly turning LSFmod into a capable HD sharpener. I also simplified, improved and optimized the code, merging preblur and secure code into the sharpening expression, and replacing preblur denoising with ex_minblur() and edge method to kroon which is both better and faster. I have to give the limit modes a second look to check how it behaves with the updates but most probably they are ok.

I'm also making a deep_resize() filter, which again simplifies+optimizes the original nnedi3_resize16(). It's one of my preferred upscalers but I didn't like it had so many dependencies specially the Dither tools and that it was slow. It won't be limited to nnedi3 but any preferred upscaler ("nnedi3", "FSRCNN", "FCBI", "SuperResXBR") because it will call ConvertFormat() internally. This allows you to simply call the kernel by its alias ("DPID","Mitchell","Catrom","Zopti"...) and ConvertFormat() will take care of everything else. I fixed some scalers implementations there and added more EWA lanczos modes by Robidoux recommendations. In order to make deep_resize() I will need to finally port limit_dif(), I will try to compare all version and find the sweet spot. That will open the doors for a debanding filter.

After that (or meanwhile) I have some ideas to develop a 1-pass scene based workflow with SC lookahead. Port some more color models (ZCAM, etc) and make an SDR to HDR filter.

[tormento]

Some summary of what I'm doing right now
What I would like to see: a white balance filter without n dependencies, aware of scene change and resilient to mostly black background with small amount of foreground enlighted objects.

And, yes, some stabilization filter inside SMDegrain, to recycle otherwise wasted vectors.

[Dogway]

What do you mean without n dependencies? GreyWorld() only requires ExTools, RgTools (might drop it) and TransformsPack.
In any case I have already the idea for SC detection that will enable scene based filtering. This will actually turn things around workflow-wise, so probably it will require adapting some of my high-level filters(?) sigh!

The current problem with GreyWorld, aside of temporal stability is that I didn't manage to align the gamut assymetry so I have to look into that a bit further. Also I probably should look for color cast/tint only on the 10% brightest percentile to have a more accurate reading (after all it's called WHITE point), but I need to test.

As for stabilization, it's a bit of a rabbit hole which I tend to avoid (like the MVTools' Zopti optimizations) so I don't get stuck in things for too long. The thing with reusing motion vectors is that you probably don't want the jittery MV for degraining, it will mostly imply creating new vectors.

I just ported latest LSFmod to EX mod, ex_limitdif() (VERY optimized) and deep_resize() (initial version but already faster (x8) and higher quality than nnedi3_resize16() ). Will upload tomorrow.

[Dogway]

Here are a few examples of deep_resize() compared to nnedi3_resize(). It's sharper while being over 4 times faster.
It's not a straight port since I added a few ideas of my own, to start with the flat resizer is now SincResize. Also the edge pass with nnedi3 which is naturally on the soft side is sharpened with LSFmod (very slightly), it's fast because of my last update. With this I noticed a big contrast change in the image, it wasn't because the sharpener but because nnedi3 and the support resizers. Well this contrast just happens to be low frequency "sharpening" akin to what you can do with ex_unsharp(), can't I undo it? Yes, I did a very small tweak to ex_unsharp() and now you can deconvolute, it gives a less coarse image while keeping high frequency sharpness. I will be uploading in the next few hours.

raw (sorry @Tormento for beating this sample lol)
http://i.imgur.com/09rWtGpm.png (https://i.imgur.com/09rWtGp.png)
spline36 (from 'Zopti' downscaled 720p)
http://i.imgur.com/pukfKorm.png (https://i.imgur.com/pukfKor.png)
nnedi3_resize16
http://i.imgur.com/PmIIxuTm.png (https://i.imgur.com/PmIIxuT.png)
deep_resize
http://i.imgur.com/ebb5TUCm.png (https://i.imgur.com/ebb5TUC.png)
deep_resize v2
http://i.imgur.com/HcAbAofm.png (https://i.imgur.com/HcAbAof.png)

spline36 (sample from FranceBB)
http://i.imgur.com/nvC3LfXm.png (https://i.imgur.com/nvC3LfX.png)
nnedi3_resize16
http://i.imgur.com/xC1qz64m.png (https://i.imgur.com/xC1qz64.png)
deep_resize
http://i.imgur.com/nbd7qdum.png (https://i.imgur.com/nbd7qdu.png)
deep_resize v2
http://i.imgur.com/sJV4EwVm.png (https://i.imgur.com/sJV4EwV.png)

[guest]

Hi Dogway, I've hit a bit of a "wall", and I don't know when what causes my problems, happened.

I have been "religiously" updating my scripts, whenever you release new ones (that I use), and I have been testing them to see if they at least pass a AVSMeter "test", but haven't been encoding with them until recently, and to my despair, I am getting errors when using certain scripts, mainly when using BM3D.

I know that SMDegrain is pretty complex, and maybe I'm missing something, especially when it comes to get BM3D to use CPU or CUDA commands.

The error is :- y4m [error]: bad sequence header magic
x264 [error]: could not open input file `-'
I have noticed in the past that this has been associated with an OpenCL problem, BUT the KNLMeansCL scripts DO work.

I have been backdating my scripts to see if I can find a point where the error doesn't occur, with no luck, yet.

I'm not sure if there's been a change with BM3D, or some of your updates have affected how BM3D is used, maybe there's more or different dependencies for recent builds.

So any suggestions would be greatly appreciated.

[Boulder]

It would be much easier to debug if you open the script in VirtualDub, or any other tool that would give the exact error message and hopefully the position as well.

[Dogway]

@TDS, with yesterday updates all the packs are now in sync and should work without issues. Normally errors will come from bugs from AVS+ x86, the plugins or the client (AVSmeter, AvsPmod...), and occasionally my fault. Just attack one at a time.
To be honest when testing BM3D a week ago I saw artifacts I didn't like, I'm not sure if it was always the case but it seemed to denoise only vertically. With that said I don't get that error, try to open in another client software to reproduce the issue. Also that hyphen error message(?) do you have strange unicode characters in your x246 output path or filename?

I will try to have a closer look later today and prepare a simple BM3D call for you to test.

[guest]

@ Boulder & Dogway, thanks for your prompt replies & suggestions...I need to remind you that I use RipBot264, and it's taken me quite a while (with help from this Forum) to incorporate these scripts into the app, and have them work successfully....I'm pretty sure I had it all running sweet several weeks ago (haven't done much for a long time, except update my scripts), so like I said earlier, I ONLY test them within RipBot264, and it uses AVSMeter to "check" the scripts, and it a little vague in "pinpointing" the actual error :(

RipBot264 uses a Distributed Encoding feature, and my problem is mainly showing up on the "remote" encoding PC's, not on the Main PC, but it's not on ALL of them, despite them all being set up the same, and there's where it becomes even more frustrating.

It's quite a time consuming process, and I'm sure you're more than aware.

Thanks

[pinterf]

anton_foy was having issues before with x86. I think there's something odd in x86 AVS+ but pinterf has to confirm, simply by testing the above Expr() call.



Ok, so 'scale_inputs' is broken on x86, I will try to ask pinterf when he's around.
Pls report bugs, I accidentally found this, so yes, there was a bug (an unaligned internal buffer) which affected LUT operation on AVX2 machines. I'm gonna make a test build later.

[Dogway]

Yes sorry had a few busy days. Aside from scale_inputs there might also an issue with 'lut' arg in Expr (ah yes what you said) and frameprops calls outside runtime environment but I can't tell because I don't use AVS x86, this is all from what users report to me. In x64 everything seems to work fine on my side.

[Fjord]

@Dogway, I'm very appreciative of your work on optimizing plugins with ExTools etc. And thanks also to all the other great contributors here!
I need some help understanding the mod versions. "EX" mods seems obvious, but what is a "MIX" mod? And why are there both EX and MIX versions of LSFmod? I don't know which I should be downloading. Or are they the same?

[Dogway]

EX mods is kind of what I would like it to be, everything run from internal functions, low dependencies.
MIX mods in turn has more dependencies as it uses RgTools, masktools2, etc

The reason I had to make MIX mods is because some operations are slower with internal code like Expr() pixel addressing being only SSSE3 accelerated vs masktools2 which uses AVX2.
ex_merge() is also generally slower than mt_merge(), and removegrain(12) or removegrain(4) are insanely faster than than ex_boxblur(mode="weighted") or ex_median("median") respectively.

In general terms MIX mods are slightly faster but output should be the same.

[StainlessS]

Maybe a Doggy glossary is in order.

[Dogway]

hahaha and I thought it was clear in OP.

[gispos]

I like to use this function (found here somewhere), but it's not the fastest. Does anyone see potential to make it faster or improve it. HBD > Dogway?

function Highlighter(clip v, float "gblur", bool "gradient", int "threshold", bool "twopass", int "amount", int "method")
{
gradient = default (gradient,true)
gblur = (gradient==true) ? default (gblur,100) : default (gblur,5.0)
threshold = default (threshold,100) #The lower the value, the more sensitive the filter will be.
twopass = default (twopass,false)
amount = default (amount,10)
method = default (method, 1) #0,1,2

amount = (amount<0) ? abs(amount) : amount

lighten= v.Tweak(sat=0).mt_lut("x "+string(threshold)+" > 0 x ?")
blurred= (gradient==true) ? lighten.gaussianblur(gblur).gaussianblur(gblur+100).gaussianblur(gblur+200) : lighten.gaussianblur(gblur)


photoshop_overlay = (method==1) ? mt_lutxy(v,blurred.invert(),"x 127.5 > y 255 x - 127.5 / * x 255 x - - + y x 127.5 / * ? ") : \
(method==2) ? mt_lutxy(v,blurred.invert(),"255 255 x - 255 y - * 255 / -") : v.Tweak(bright=amount)

photoshop_overlay = (twopass==true) ? photoshop_overlay.SmoothLevels(gamma=2.0, Lmode=2) : photoshop_overlay
return mt_merge(v,photoshop_overlay,blurred)
}

[Dogway]

Yes, big potential, give me some minutes.

EDIT: Here it is. By the way, what is it for?

function Highlighter(clip v, float "gblur", bool "gradient", int "threshold", bool "twopass", int "amount", int "method") {

bi = BitsPerComponent(v)
fs = propNumElements (v,"_ColorRange") > 0 ? \
propGetInt (v,"_ColorRange") == 0 : false

grad = Default (gradient,true)
gblur = Default (gblur, grad ? 100 : 5)
thres = Default (threshold,100) #The lower the value, the more sensitive the filter will be.
twopass = Default (twopass,false)
amount = Default (amount,10)
method = Default (method, 1) # 0, 1 or 2

amount = ex_bs(abs(amount), 8, bi, true)
thres = ex_bs(thres, 8, bi, true)

lighten = v.ex_lut(Format("x {thres} > 0 x ?"), UV=128)
blurred = grad ? lighten.ex_gaussianblur(0.073*gblur+17.78,UV=1) : \
lighten.ex_gaussianblur(0.064*gblur+ 3.75,UV=1)


photoshop_overlay = (method==1) ? ex_lutxy(v,blurred,"x range_half > range_max y - Y@ range_max x - X@ range_half / * range_max + Y x range_half / * ? ", fulls=fs) : \
(method==2) ? ex_lutxy(v,blurred,"range_max range_max x - y * range_max / -",fulls=fs) : v.ex_lut(Format("x {amount} + ")+(!fs?"ymax min":""), fulls=fs)

photoshop_overlay = twopass ? photoshop_overlay.ex_contrast(-0.8,1,167,tv_range=!fs) : photoshop_overlay
return mt_merge(v,photoshop_overlay,blurred) }

[gispos]

Yes, big potential, give me some minutes.

EDIT: Here it is. By the way, what is it for?


To brighten up dark scenes (Maybe wrongly worded, 'dark areas' is probably more correct).
Will test your changes with a 'Preview Filter'. :D
Thank you!

[gispos]

What am I missing?

Script error: There is no function named 'nmod'.
(D:\Tools\AviSynth\plugins64\DogwayFilters\ExTools.avsi, line 1691)

Found:
Located in ResizerPack.avsi
Hm, wouldn't it make more sense to put it in ExTools.avsi.

I don't really want to have to load all your packs... that's too much 'resources' for me if I only want to use one function 'ex_Highlighter'

Now I get:

Expr: Failed to convert '{amount}' to float
(D:\Tools\AviSynth\plugins64\DogwayFilters\ExTools.avsi, line 67)

[Dogway]

Yes sorry, I forgot to add Format() to the expression (edited), I got a phone call in between and got distracted.
Don't be like that, it's only 5 packs, I really don't like scripts that are like a junkyard where you put everything in it with no correlation.
ExTools is only for core filters, low level or build block functions. I only try to be conservative at big names like SMDegrain or QTGMC.
Give the functions a try, mmod(), PadBorders(), deep_resize(), you will surely get in the loop and won't come back :P

[gispos]

Hi Dogway,
the 'Highlighter' filter is now called 'ex_Highlighter' and is 6 - 8 times faster with almost identical results. So full success.

And therefore (if you feel like it) you may try the opposite function. 'HighlightLimiter' and 'HighlightLimiterMod'
The filter reduces too bright areas. The mod version was changed by me (long time ago).

function HighlightLimiter(clip v, float "gblur", bool "gradient", int "threshold", bool "twopass", int "amount", int "method")
{
gradient = default (gradient,true) #True uses the gaussian blur to such an extent so as to create an effect similar to a gradient mask being applied to every area that exceeds our threshold.
gblur = (gradient==true) ? default (gblur,100) : default (gblur,5.0) #The strength of the gaussian blur to apply.
threshold = default (threshold, 160) #The lower the value, the more sensitive the filter will be.
twopass = default (twopass, false) #Two passes means the area in question gets darkened twice.
amount = default (amount, 20) #The amount of brightness to be reduced, only applied to method=2
method = default (method, 0) #0,1 Method 0 is multiply, the classic HDR-way. Any other method set triggers a brightness/gamma approach.

amount = (amount>0) ? -amount : amount

darken=v.Tweak(sat=0).mt_lut("x "+string(threshold)+" < 0 x ?")
blurred= (gradient==true) ? darken.gaussianblur(gblur).gaussianblur(gblur+100).gaussianblur(gblur+200) : darken.gaussianblur(gblur)
multiply = (method==0) ? mt_lut(v,"x x * 255 /") : v.Tweak(bright=amount)
multiply = (method==0) ? eval("""
(twopass==true) ? mt_lutxy(multiply,v,"x y * 255 /") : multiply""") : eval("""
(twopass==true) ? multiply.SmoothLevels(gamma=0.9, Lmode=2) : multiply""")

return mt_merge(v,multiply,blurred)
}

function HighlightLimiterMod(clip clp, Int "threshold", Int "dark", Int "rad", int "mode", float "gamma1_2"){
threshold=default(threshold, 160)
dark=default(dark, 255)
rad=default(rad, 64)
mode=default(mode, 0) # 0,1,2
gamma1_2=default(gamma1_2, 0.6) # for mode 1,2

m = clp.Width() >= clp.Height() ? clp.Height() : clp.Width()
rad = round(m/rad/4) < 1 ? max(round(m/4.0),16): rad
darken=clp.Tweak(sat=0).mt_lut("x "+string(threshold)+" < 0 x ?")
blurred=darken.bicubicresize(round(clp.width()/rad/4)*4, round(clp.height()/rad/4)*4).bicubicresize(clp.width(),clp.height(),1,0)
multiply = (mode==0) ? mt_lut(clp,"x x * " + string(dark) + " /") : (mode==1) ? SmoothLevels(clp, gamma=gamma1_2) : mt_lut(clp,"x x * " + string(dark) + " /").SmoothLevels(gamma=gamma1_2)
return mt_merge(clp,multiply,blurred)
}

Thanks already

[guest]

Hey Dogway, and the updates just keep on coming...

So I just noticed in the latest iteration of SMDegrain, that KNLMeansCL (pre filter=5) has "disappeared" !!!!

In it's place, has appeared DGDecNV !!!! How does that work.

Can "we" have KNLMeansCL back ???

I haven't done anymore testing BM3D since my post a few days ago (been busy with other things), I will post my script & call's soon, and I would appreciate your help with that.

Have the dependencies for BM3D changed recently ??

[kedautinh12]

You can try TemporalDegrain2 (postFFT=4), that option have KNLMEANSCL :D

[guest]

You can try TemporalDegrain2 (postFFT=4), that option have KNLMEANSCL :D

Hi ked,

But why remove it altogether...just add another pre filter (DGDecNV).

I'm having enough issues (see post 783), that I haven't really had time to "fix", and then sort of unannounced there is a significant change to the pre-filters in SMDegrain :(

[Dogway]

Hey Dogway, and the updates just keep on coming...

So I just noticed in the latest iteration of SMDegrain, that KNLMeansCL (pre filter=5) has "disappeared" !!!!

In it's place, has appeared DGDecNV !!!! How does that work.

Can "we" have KNLMeansCL back ???

I haven't done anymore testing BM3D since my post a few days ago (been busy with other things), I will post my script & call's soon, and I would appreciate your help with that.

Have the dependencies for BM3D changed recently ??

Don't worry the updates are almost done, I will be working mainly on Zopti (SMDegrain, QTGMC, FRC, etc) and SceneStats for the following weeks.

Do you have something to prove KNLMeansCL should be back? In my tests it showed poor performance, see the following images (very bad motion estimation).

KNLMeans Spatio-Temporal (bug so edited out)
KNLMeans Spatial (bug so edited out)
DGDenoise
http://i.imgur.com/0Wqxc1Km.png (https://i.imgur.com/0Wqxc1K.png)
DGDenoise has sharp edges so a bit of blurring afterwards is fine, I will add it.

As for the BM3D issues I would first check BM3D works fine on its own so it's not an ex_BM3D() bug.

[ChaosKing]

Results are a bit strange. Isn't DGDenoise also an implementation of nlmeans algo? (but without any temporal filtering)
I mean they should look more similar...

Is your source filter frame accurate?

[Dogway]

I'm using DGDecNV r239 so I it should be frame accurate.
I'm not sure about DGDenoise algorithm, there are several edge aware denoisers like nlmeans, bilateral and variations like smart blur, surface blur and those from the papers; guided blur, etc. To me DGDenoise looks like a bilateral filter but videoh has to confirm.
Temporal GPU denoisers in AviSynth is a chimera, both KNLMeansCL and BM3D have issues BM3D CPU is fine though.

[ChaosKing]

From DGDecodeNVManual.html
DGDenoise() is a fast stand-alone CUDA filter that implements NLM denoising. The input clip must be in the YV12 or YUV420P16 format. DGSource() delivers YUV420P16 when the source stream has a 10/12/16-bit depth.

I never had such blending artifacts in knlm temporal. But I remember that some graphics card had some issues with this plugin. Do you get the same output in vapoursynth too?

[guest]

Don't worry the updates are almost done, I will be working mainly on Zopti (SMDegrain, QTGMC, FRC, etc) and SceneStats for the following weeks.

Do you have something to prove KNLMeansCL should be back? In my tests it showed poor performance, see the following images.

Very bad motion estimation. If converted to spatial only, not as good flat denoiser as DGDenoise and edge blurring in excess. DGDenoise has sharp edges so a bit of blurring afterwards is fine, I will add it.

As for the BM3D issues I would first check BM3D works fine on its own so it's not an ex_BM3D() bug.

I have no "good" reason to want KNLMeansCL back, I guess it's just that when you set up certain scripts to use a particular pre-filter, and then it changes without warning, that's the only reason, I guess,
if you think that DG's stuff works better, then so be it.

You also hinted that there are some issues with BM3D (GPU), but with BM3D (CPU) it's ONLY good for AVX2 CPU's, unless the dev of that updates that function.

as for this As for the BM3D issues I would first check BM3D works fine on its own so it's not an ex_BM3D() bug. not sure how to do that.

All I can say about that is, that it used to work, but since updating last week sometime, the error showed up, but I can't figure out what versions I changed, that brought it back.

I need to back date, and start almost from scratch :(

BTW, where can I get older builds ??? is there an archive somewhere ??

[Dogway]

From DGDecodeNVManual.html

I never had such blending artifacts in knlm temporal. But I remember that some graphics card had some issues with this plugin. Do you get the same output in vapoursynth too?

Oh :scared:! ::facepalm:: shameful bug. Will update later today and reimplement ex_KNLMeansCL (happy TDS)

@TDS: Wait for v3.4.6 tonight or something and report back if it still doesn't work.

[guest]

Oh :scared:! ::facepalm:: shameful bug. Will update later today and reimplement ex_KNLMeansCL (happy TDS)

@TDS: Wait for v3.4.6 tonight or something and report back if it still doesn't work.

Well it looks as though you've left all the KNLMeansCL stuff in SMDegrain, so why not leave that as pre filter 5, and put in the new one as either 6 or 7, well probably better at 7, and leave bm3d as 6.

[kedautinh12]

Well it looks as though you've left all the KNLMeansCL stuff in SMDegrain, so why not leave that as pre filter 5, and put in the new one as either 6 or 7, well probably better at 7, and leave bm3d as 6.

Doggy updated latest ver :D

[guest]

Doggy updated latest ver :D

Thnx ked :)

@Dogway, nice work, I see what you've done there....will "test" as soon as I can, and report back.

Cheers

---------------------------------------------------------------------------

So, reporting back, and have had limited success, it works on a stand alone PC, but using the Distributed Encoding feature of RipBot264, the Client encoding servers will not encode :(, with this error y4m [error]: bad sequence header magic
x264 [error]: could not open input file `-' OR y4m [error]: bad sequence header magic
x265 [error]: could not open input file `-'

I have no idea wtf has gone wrong, 'cause I'm sure it was working a couple of weeks ago before I "blindly" updated everything :(

---------------------------------------------------------------------------

Another update, I have managed to get a "set" of scripts from and old build of RipBot264, which has most of your "goodies", but rather old builds, BUT those errors have gone :)

So now I need to update one @ a time, and see where the issue is / was. I'm kinda thinkin' it might be something to do with BM3D, but KNL did the same thing, suspecting some OpenCL problem, but wtfk !

[tormento]

both KNLMeansCL and BM3D have issues BM3D CPU is fine though.
I dunno.

I have always used both with a bias for the second because of more detail retaining while doing a good job at helping removing noise.

I have never found any artifact in the output video from SMDegrain and I will keep on using BM3D on CUDA, as my CPU is more ancient than simply old. :)

DGDenoise is a nice add, I still have to evaluate detail retaining and differences with KNLMeansCL.

[Dogway]

With BM3D you have to be too careful on many aspects. First don't crop with DGSource(). And second, we lack "blocking_step" arg, so the maximum block-free settings are sigma=4, radius=1, which is almost useless for prefiltering. I think it also has issues with FFMS2 loader. With CUDA=false I didn't see any of these bugs.

DGDenoise() is fine for Anime, as it's only spatial filtering. I haven't run benchmarks though, it would be very interesting to compare DGDenoise() and KNLMeansCL().

[tormento]

With BM3D you have to be too careful on many aspects. Fist don't crop with DGSource().
I always do. How could it interfere? Please provide some examples.
it would be very interesting to compare DGDenoise() and KNLMeansCL().
I haven't done any serious comparison yet but SMDegrain is way faster when using DGDenoise instead of KNLMeansCL. About denoising, using DGDenoise as prefilter, gives bitrates that are between KNLMeansCL (the lowest ones but with detail loss) and BM3D, that AFAIK was better preserving details, mostly moving ones.

[Dogway]

I posted the issue here (https://forum.doom9.org/showthread.php?p=1963046#post1963046). Confirmed by other users.

DGDenoise is purely spatial, you would need to compare with spatial KNLMeansCL as it's spatio-temporal in SMDegrain, and do some matching with strengths. For visual comparison you can use show=true or directly compare the filters outside SMDegrain.

EDIT: I made a benchmark between both.

ex_DGDenoise(str=0.10,UV=3,LFR=0) # P(4) 83
ex_KNLMeansCL(a=2,s=2,d=0,h=6.0,wmode=1,chroma=true,device_type="GPU",LFR=0) # P(4) 80

DGDenoise is very similar to KNLMeansCL when used spatially with wmode=1 which retains a bit more of the lineart. But suprisingly KNLMeansCL retains more low frequency detail than DGDenoise, this might be good or bad as it will also show faint blotches of grain.

[Dogway]

Hi Dogway,
the 'Highlighter' filter is now called 'ex_Highlighter' and is 6 - 8 times faster with almost identical results. So full success.

And therefore (if you feel like it) you may try the opposite function. 'HighlightLimiter' and 'HighlightLimiterMod'
The filter reduces too bright areas. The mod version was changed by me (long time ago).

Check this:

function HighlightLimiter(clip v, float "gblur", bool "gradient", int "threshold", bool "twopass", int "amount", int "method") {

bi = BitsPerComponent(v)
fs = propNumElements (v,"_ColorRange") > 0 ? \
propGetInt (v,"_ColorRange") == 0 : false

grad = Default (gradient,true)
gblur = Default (gblur, grad ? 100 : 5)
thres = Default (threshold,160) #The lower the value, the more sensitive the filter will be.
twopass = Default (twopass,false)
amount = Default (amount,20)
method = Default (method, 0) # 0 or 1

amount = ex_bs(abs(amount), 8, bi, true)

darken = v.ex_binarize(thres,hi=-1)
blurred = grad ? darken.ex_gaussianblur(0.073*gblur+17.78,UV=1) : \
darken.ex_gaussianblur(0.064*gblur+ 3.75,UV=1)

multiply = (method==0) ? v.ex_lut("x dup * range_max /", fulls=fs) : v.ex_lut(Format("x {amount} + ")+(!fs?"ymax min":""), fulls=fs)
multiply = !twopass ? multiply : \
(method==0) ? multiply.ex_lutxy(v,"x y * range_max /", fulls=fs) : \
multiply.ex_contrast(0.15, 1, 163, tv_range=!fs)
return mt_merge(v,multiply,blurred) }


function HighlightLimiterMod(clip v, int "threshold", int "dark", float "rad",int "mode", float "gamma1_2") {

w = Width (v)
h = Height(v)
bi = BitsPerComponent(v)
fs = propNumElements (v,"_ColorRange") > 0 ? \
propGetInt (v,"_ColorRange") == 0 : false

thres = Default (threshold,160)
dark = Default (dark, 255)
rad = Default (rad, 64)
mode = Default (mode, 0) # 0,1,2
gamma1_2= Default (gamma1_2, 0.6) # for mode 1,2

m = min(w,h)
dark = ex_bs(dark, 8, bi, true)
rad = round(m/rad/4) < 1 ? max(round(m/4.0),16) : rad

darken = v.ex_binarize(thres,hi=-1)
blurred = darken.ex_GaussianBlur(rad*0.83825-0.0631,pad=true,UV=1)

multiply = (mode==0) ? v.ex_lut(Format("x dup * {dark} /"), fulls=fs) : \
(mode==1) ? v.ex_levels(gamma=gamma1_2,tv_range=!fs) : \
v.ex_lut(Format("x dup * {dark} /"), fulls=fs).ex_levels(gamma=gamma1_2,tv_range=!fs)

return mt_merge(v,multiply,blurred) }

[guest]

Hey Dogway,

I've had a major win (several win's, actually).

So I started from an old "set" of your scripts, and worked thru until they worked.

I even got the DGDecNV pre-filter to work :)

So this is my script for KNLMeansCL, is there any "fine tuning" that could be done to it, as I noticed you changed the instructions a bit in SMDegrain 3.4.6d

LoadPlugin("%AVISYNTHPLUGINS%\masktools\masktools2.dll")
LoadPlugin("%AVISYNTHPLUGINS%\MedianBlur2\MedianBlur2.dll")
LoadPlugin("%AVISYNTHPLUGINS%\mvtools\mvtools2.dll")
LoadPlugin("%AVISYNTHPLUGINS%\KNLMeansCL\KNLMeansCL.dll")
LoadPlugin("%AVISYNTHPLUGINS%\RgTools\RgTools.dll")
Import("%AVISYNTHPLUGINS%\scripts\extras\ExTools74.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\LSFmod50ex.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Resizers Pack78.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Sharpeners Pack34.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\SMDegrain346d.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Zs_RF_Shared.avs")
pre=ex_KNLMeansCL(video,a=2,s=4,d=1,h=4,wmode=0)
video=SMDegrain(video,tr=4,thSAD=400,thSADC=200,thSCD2=80,contrasharp=true,prefilter=pre,refinemotion=true)

And this is my DGDecNV script, again, any "fine tuning" for this :-

LoadPlugin("%AVISYNTHPLUGINS%\dgdecnv\DGDecodeNV.dll")
LoadPlugin("%AVISYNTHPLUGINS%\dfttest\dfttest.dll")
LoadPlugin("%AVISYNTHPLUGINS%\masktools\masktools2.dll")
LoadPlugin("%AVISYNTHPLUGINS%\MedianBlur2\MedianBlur2.dll")
LoadPlugin("%AVISYNTHPLUGINS%\mvtools\mvtools2.dll")
LoadPlugin("%AVISYNTHPLUGINS%\RgTools\RgTools.dll")
Import("%AVISYNTHPLUGINS%\scripts\extras\ExTools74.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\LSFmod50ex.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Sharpeners Pack34.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\SMDegrain346d.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Transform Packv1.0 RC36.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Zs_RF_Shared.avs")
pre=ex_DGDenoise(video,str=0.10)
video=SMDegrain(video,tr=2,thSAD=400,thSADC=100,thSCD2=80,contrasharp=true,prefilter=pre,refinemotion=true)

HOWEVER, can you please change SMDegrain back to it's BM3D default (earlier builds) being CUDA, you've done something and now it's BM3D CPU by default, and I need CUDA for some of my older servers that don't support BM3D CPU builds.

It was fairly easy to change the CUDA settings from "True" to "False", but with this latest build, that doesn't seem to apply. :(

Thanks

[kedautinh12]

Hey Dogway, i added some ex_ family into f3kgrain. Can you check it's complete replace ex_ into f3kgrain or not?? Thanks
https://github.com/kedaitinh12/AVSPlus-Scripts/blob/master/f3kgrain_v0.4MOD.avsi

[Dogway]

Hey Dogway, i added some ex_ family into f3kgrain. Can you check it's complete replace ex_ into f3kgrain or not?? Thanks
https://github.com/kedaitinh12/AVSPlus-Scripts/blob/master/f3kgrain_v0.4MOD.avsi

I reply you there.

@TDS: You should be able to use CUDA by setting it to true here (https://github.com/Dogway/Avisynth-Scripts/blob/9b84fe4957f5f748d85309b6ef664d634d4ca6f9/SMDegrain%20v3.4.6d/SMDegrain%20v3.4.6d.avsi#L248).

For the KNLMeansCL prefilter, test with wmode=1, depending on content you might like it more. By default with 'd' > 0, LFR is used, you can try disabling it with LFR=0, but it will smear quite a bit. ex_DGDenoise() seems also to smear quite a bit, so for next version I will also add LFR argument.
CUDA in BM3D is broken really, you can try comparing them CUDA vs CPU outside SMDegrain, I posted this in the BM3D thread.

[ChaosKing]

And second, we lack "blocking_step" arg, so the maximum block-free settings are sigma=4, radius=1, which is almost useless for prefiltering.

Not true, there is a blocking_step paramater in the cuda version. I just set mine to 2 so I don't have to worry about evil blocking :devil:

[Dogway]

Oh shii... the AVS filter is so undocumented it looks like shady business haha

block_step, that is: https://github.com/WolframRhodium/VapourSynth-BM3DCUDA/blob/94e1380cab621b9a9226824e7ba6876b8b502c78/source/kernel.cu#L49

[anton_foy]

Oh shii... the AVS filter is so undocumented it looks like shady business haha

block_step, that is: https://github.com/WolframRhodium/VapourSynth-BM3DCUDA/blob/94e1380cab621b9a9226824e7ba6876b8b502c78/source/kernel.cu#L49

Tried out BM3DCUDA again two days ago and also lowered block_step to 2 which rid the blocking.
I usually go here (http://avisynth.nl/index.php/BM3DCUDA)

[ChaosKing]

Undocumented? It's in the paramaters list here https://github.com/WolframRhodium/VapourSynth-BM3DCUDA#parameters
It is mentioned 3 times in the read me ;) (ok, note2 says blocking_step instead of block_step)

[Dogway]

There's also bm_range and ps_range so I made a preset matrix to conform to original profiles. For preset "fast" and "normal" I will lower the array by 1, I had to make an ArrayOp() function, just edited the old MatrixMul() in TransformsPack so now it's multipurpose.

[gispos]

Check this:

function HighlightLimiter(clip v, float "gblur", bool "gradient", int "threshold", bool "twopass", int "amount", int "method") {

bi = BitsPerComponent(v)
fs = propNumElements (v,"_ColorRange") > 0 ? \
propGetInt (v,"_ColorRange") == 0 : false

grad = Default (gradient,true)
gblur = Default (gblur, grad ? 100 : 5)
thres = Default (threshold,160) #The lower the value, the more sensitive the filter will be.
twopass = Default (twopass,false)
amount = Default (amount,20)
method = Default (method, 0) # 0 or 1

amount = ex_bs(abs(amount), 8, bi, true)

darken = v.ex_binarize(thres,hi=-1)
blurred = grad ? darken.ex_gaussianblur(0.073*gblur+17.78,UV=1) : \
darken.ex_gaussianblur(0.064*gblur+ 3.75,UV=1)

multiply = (method==0) ? v.ex_lut("x dup * range_max /", fulls=fs) : v.ex_lut(Format("x {amount} + ")+(!fs?"ymax min":""), fulls=fs)
multiply = !twopass ? multiply : \
(method==0) ? multiply.ex_lutxy(v,"x y * range_max /", fulls=fs) : \
multiply.ex_contrast(0.15, 1, 163, tv_range=!fs)
return mt_merge(v,multiply,blurred) }


function HighlightLimiterMod(clip v, int "threshold", int "dark", float "rad",int "mode", float "gamma1_2") {

w = Width (v)
h = Height(v)
bi = BitsPerComponent(v)
fs = propNumElements (v,"_ColorRange") > 0 ? \
propGetInt (v,"_ColorRange") == 0 : false

thres = Default (threshold,160)
dark = Default (dark, 255)
rad = Default (rad, 64)
mode = Default (mode, 0) # 0,1,2
gamma1_2= Default (gamma1_2, 0.6) # for mode 1,2

m = min(w,h)
dark = ex_bs(dark, 8, bi, true)
rad = round(m/rad/4) < 1 ? max(round(m/4.0),16) : rad

darken = v.ex_binarize(thres,hi=-1)
blurred = darken.ex_GaussianBlur(rad*0.83825-0.0631,pad=true,UV=1)

multiply = (mode==0) ? v.ex_lut(Format("x dup * {dark} /"), fulls=fs) : \
(mode==1) ? v.ex_levels(gamma=gamma1_2) : \
v.ex_lut(Format("x dup * {dark} /"), fulls=fs).ex_levels(gamma=gamma1_2)

return mt_merge(v,multiply,blurred) }
Thanks Dogway. Results:
For HighlightLimiter, your version 'ex_HighlightLimiter' wins. Almost the same result but 10 times faster.

I can't decide about HighlightLimiterMod.
Your version is 8% faster but there are small differences. Sometimes darker areas are also slightly darkened.
Which parameters could I play around with?

On these?
blurred = darken.ex_GaussianBlur(rad*0.83825-0.0631,pad=true,UV=1)

Edit: Everything Ok. I just have to make the radius a little smaller to get almost the same result.

[Dogway]

8% or 8 times :rolleyes: ?

Yes, mapping bicubic blur to gaussian blur is not exact but the later gives better results gaussian like, I couldn't decide if this or (rad-1) so tweak visually.


EDIT: By the way, I made a bandpass replacement filter to replace mid frequencies from one clip to another, please have a look if you find something odd. I think gaussian is not ideal here (might need mean blur) but anyway here it is:


# Bandpass replacement
# Replace Mid Frequencies of first clip from the second clip
#
# Bandpass workflow diagram (shaded '/' means deleted by lowpassing):
#
# A A A A A
# _______ _______ _______ _______ _______
# | | |/ / / /| | | |/ / / /| | |
# | | | / / / | | | | / / / | | |
# | | |-------|-> HI ------- |/ / / /| -------
# | | | | | / / / |
# | | - | | = + |/ / / /| =
# | | | | |-------|-> LO _______
# | | | | | | | |
# | | | | | | | |
# ------- ------- ------- -------
# A
# B B B / A+B
# _______ _______ / _______
# |/ / / /| |/ / / /| /\ + | |
# | / / / | | / / / | / \ | A |
# |-------|-> HI |/ / / /| _______ / \ |-------|
# | | | / / / | | | B -----> | |
# | | - |/ / / /| = | | | B |
# | | |-------|-> LO ------- |-------|
# | | | | | |
# | | | | | A |
# ------- ------- -------
#
# Example:
# ex_MFR(a,b,1700,700) # Replace mid frequencies from 700 to 1700 from 'b' to 'a'
#
#
function ex_MFR (clip c, clip ref, float "hi", float "lo", int "UV") {

rgb = isRGB (c)
is42 = is420 (c)
w = width (c)
h = height(c)
mx = max(w,h)
bi = BitsPerComponent(c)
fulls= propNumElements (c,"_ColorRange") > 0 ? \
propGetInt (c,"_ColorRange") == 0 : rgb

HI = Default( HI, 1200*(w/1920.)) # Default is 1200 for 1080p
LO = Default( LO, HI/2. ) # Range of Mid Frequency to restore
UV = Default( UV, 3) # Set UV to 4 to copy the chroma from the 'ref' clip
UV2 = UV==4?2:UV

LO = max(LO, 4)

Fs = max(w,h) * 2 # Frequency sample rate is resolution * 2 (for Nyquist)
k = sqrt(log(2)/2) * HI # Constant for -3dB
HI = Fs / ( k * 2 * pi ) # Frequency Cutoff for Gaussian Sigma

k = sqrt(log(2)/2) * LO
LO = Fs / ( k * 2 * pi )

LOUV = c.is422() || is42 ? LO/2. : c.isYV411() ? LO/4. : LO
LOUVv= is42 ? LO/2. : LO
HIUV = c.is422() || is42 ? HI/2. : c.isYV411() ? HI/4. : HI
HIUVv= is42 ? HI/2. : HI

mpc = c.vsTCanny(HI, HIUV, HIUVv, mode=-1, u=UV2, v=UV2)
mpc2 = c.vsTCanny(LO, LOUV, LOUVv, mode=-1, u=UV2, v=UV2)
mpr = ref.vsTCanny(HI, HIUV, HIUVv, mode=-1, u=UV2, v=UV2)
mpr2 = ref.vsTCanny(LO, LOUV, LOUVv, mode=-1, u=UV2, v=UV2)

Expr(c, mpc, mpc2, mpr, mpr2, "x y - z + a b - +", isy(c) ? Undefined() : ex_UVexpr("x y - z + a b - +", UV2, bi, rgb, fulls, "none") ) }

[gispos]

8% or 8 times :rolleyes: ?
Not a typo, 8 to 10% is already correct.

The normal version used:
blurred=darken.gaussianblur(gblur).gaussianblur(gblur+100).gaussianblur(gblur+200)

and that is slower than my Mod version:
blurred=darken.bicubicresize(round(clp.width()/rad/4)*4, round(clp.height()/rad/4)*4).bicubicresize(clp.width(),clp.height(),1,0)

therefore the difference to your variant is not so big:
blurred = darken.ex_GaussianBlur(rad*0.83825-0.0631,pad=true,UV=1)


EDIT: By the way, I made a bandpass replacement filter to replace mid frequencies from one clip to another, please have a look if you find something odd.
Um, I do not know what to do with it? There are no reference clips for my clips. Surely this must be the same content?

Edit:
I think the topic of lightening dark areas or reducing light areas interests many users here. One could prepare with it here many a joy.:)

Edit2:
The normal version you changed 'ex_HighlightLimiter' is the best, same results but 8 to 10 times faster. Thanks again.

[Dogway]

Yes I noticed, you were already using bicubic for blurring which is inherently fast, you can also try vsTCanny for blurring, for mid sigmas sometimes it's faster than ex_gaussianblur().
I edited the scripts because I missed 'tv_range' in ex_levels. Well 10% is already very good in my book. These filters look to me like mini Retinex filters, the concept works fine, I will have to compare to ex_retinex() for speed and quality benchmarks.

No, ex_MFR() is totally unrelated, I was testing on improving UVRecon in TransformsPack finally used ex_LFR though.

Also for anyone interested, I'm making a window kernel list in Desmos (https://www.desmos.com/calculator/qs2j9oo7ku) for study (edited from here (https://www.reddit.com/r/DSP/comments/ojlae6/i_made_a_desmos_graph_that_demonstrates_various/)) and include in TransformsPack, still investigating on how to derive 'taps'.

[gispos]

Dogway, one more question:
You use the properties for TV_Range, I don't really like that. At least I still use filters that don't pass properties.
I have added tv_range as parameter and and also changed the 'fs' assignments in the code. Do you see any disadvantages?

function ex_Highlighter(clip v, float "gblur", bool "gradient", int "threshold", bool "twopass", int "amount", int "method", bool "tv_range") {

bi = BitsPerComponent(v)
fs = default(tv_range, True)

grad = Default (gradient,true)
gblur = Default (gblur, grad ? 100 : 5)
thres = Default (threshold,100) #The lower the value, the more sensitive the filter will be.
twopass = Default (twopass,false)
amount = Default (amount,10)
method = Default (method, 1) # 0, 1 or 2

amount = ex_bs(abs(amount), 8, bi, true)
thres = ex_bs(thres, 8, bi, true)

lighten = v.ex_lut(Format("x {thres} > 0 x ?"), UV=128)
blurred = grad ? lighten.ex_gaussianblur(0.073*gblur+17.78,UV=1) : \
lighten.ex_gaussianblur(0.064*gblur+ 3.75,UV=1)

photoshop_overlay = (method==1) ? ex_lutxy(v,blurred,"x range_half > range_max y - Y@ range_max x - X@ range_half / * range_max + Y x range_half / * ? ", fulls=!fs) : \
(method==2) ? ex_lutxy(v,blurred,"range_max range_max x - y * range_max / -",fulls=!fs) : v.ex_lut(Format("x {amount} + ")+(fs?"ymax min":""), fulls=!fs)

photoshop_overlay = twopass ? photoshop_overlay.ex_contrast(-0.8,1,167,tv_range=fs) : photoshop_overlay
return mt_merge(v,photoshop_overlay,blurred)
}

[Dogway]

'fulls' should have the opposite sign of 'tv_range' so I think it would make better sense to write "fs = !Default(tv_range, True)", then change according in following calls (whether the filter expects 'fulls' or 'tv_range').
But what I normally do is "tv = Default(tv_range, True)" and use 'tv' or '!tv' upon requirements.

Yes, filters that don't pass frameprops is a problem, maybe we should make a list of them. I have been thinking for some time as a header filter to define initial frameprops aside those assigned by the loader, so you can add _ColorRange, or override other frameprops. I have also been thinking on a color space detection but I will leave that for the future.

[gispos]

Hello Dogway,
I just downloaded your latest version.

Very nice that you have already added the AvsPmod function definetion to many functions.
Also the histogram curves are very nice (especially with 'Preview Filter :)).

A wish from me to ex_contrast:
Is it possible to add another pivot slider? Or something like a 'soft limiter'.
So that you can delimit dark, middle and light areas more.

What I noticed is that the contrast value in the middle range does not allow any 0.01 changes, only from 0.1 can visible changes of 0.01 be made.
This is probably due to mathematical facts and cannot be improved?

I just don't like setting parameters by reading the frame properties. This should not be done in hidden functions
Why not set 'tv_range' manually? But this is only my opinion.

[Dogway]

Yes, with the preview histograms and your preview filters this is starting to look like a decent alternative to more modern packages. Just tested Preview Filters and it's awesome, one problem though, I have to add 'last' to my script after '**/' or AvsPmod fails to render output (in case my preview filter is the last one in the script).

I didn't understand your question. You can play with 'pivot' to have dark or light areas affect more, for middle you have gamma in ex_levels (not exactly unbiased but mostly).

ex_contrast function definition is "by 0.1", does it work if you change it to "0.01"? I see changes in the histogram curve going from 0.0 to 0.01 and all the way up to 0.1.

I had a few wishes for function definition, one is max stepping for int types is '4', I don't know why. The other is string types doesn't understand anything that is not alphanumeric, so "string preset="[fast]" ("[slow]" / "[fast]") " fails.

As for frameprops, in my functions they are only read as a fallback (Defaults) if you explicitly don't define the argument, so no problem with that.

[gispos]

I didn't understand your question. You can play with 'pivot' to have dark or light areas affect more, for middle you have gamma in ex_levels (not exactly unbiased but mostly).

ex_contrast function definition is "by 0.1", does it work if you change it to "0.01"? I see changes in the histogram curve going from 0.0 to 0.01 and all the way up to 0.1.

There is a difference whether I can adjust the curve with only one or two controllers and thus have, for example, a control value that can be adjusted from the dark to only the middle areas. SmoothLevels, for example, has extra controls for dark and light areas and the mode parameter can be used to adjust the behavior.

It depends on how you specify the slider values (I do not use your examples). Values from 0.01 are set visible, but only after the value has at least 0.1
From 0.00 to 0.01 there is a jump as if 0.1 was set.


I had a few wishes for function definition, one is max stepping for int types is '4', I don't know why.
I don't know that either. But it should depend on the maximum value of the slider. Must see if I can find that.


The other is string types doesn't understand anything that is not alphanumeric, so "string preset="[fast]" ("[slow]" / "[fast]") " fails.

I think it's because there are valid rules for self-created sliders. And [ ] is for example responsible as a character for slider start and end.
At that time there were e.g. also no arrays in avisynth. The whole mesh around the slider is far-reaching and requires time, effort and desire to tackle.
Edit: In the function definitions [ ] are a flag for hidden parameters. Have a look at the filterdb.dat. The whole thing must also be readable and implementable, so there are certain rules.


As for frameprops, in my functions they are only read as a fallback (Defaults) if you explicitly don't define the argument, so no problem with that.
Ok.

AvsPmod fails to render output (in case my preview filter is the last one in the script).
Sometimes I also have to use a return last in the script. These are the requirements avisynth has for its syntax.
I can't check with every parameter change if a return last is present and also at a valid position. You will manage that!

[gispos]

I had a few wishes for function definition, one is max stepping for int types is '4', I don't know why.

It works for me. max and min and default values must of course be divisible by e.g. 5 or must be 0

There is even a warning if you do not meet these requirements.
I just defined a slider with the values: min 0 default 100 max 200 step by 50

[ENunn]

Does anyone know why Avisynth has been more unstable since I installed the scripts? I'm trying to use QTGMC and the moment I try to load the preview Avspmod insta-crashes. Ffmpeg does nothing and crashes as well. Before I kept getting access violation errors with ZS_RF_Shared.avsi, either that or it would load fine when it feels like it. Now it's not loading at all. I have the latest version of the script and the dependencies so I'm not sure what's up.

[gispos]

I try to load the preview Avspmod insta-crashes.
What version of Avisynth are you using?

[ENunn]

What version of Avisynth are you using?

3.7.1.

I cleared out my plugins folder and installed them all back in, looks like neo_fft3d.dll might be the culprit for the crashing, which kinda sucks because I use tempdegrain2 a lot. But now I'm getting random access violations with ExTools.avsi.

[guest]

So my BM3D woe's continue.

I have several older PC's that aren't AVX2, BUT they have reasonable nVidia GPU's for CUDA / OpenCL support.

Naturally BM3D CPU doesn't work, BUT I just can't get BM3D CUDA to work on them. I really thought I got it to work before...but can't get it to, anymore :(

KNLMeansCL & DGDenoise work fine.

So do I just forget about using BM3D pre-filters ???, what's the next best thing ??

I'm tearing my hair out (and I haven't got much of that anyway) !!

Help...

[Dogway]

@TDS: I think your issue would probably be one of resources distribution through the server. Maybe CUDA instructions are not passed, you can try first outside avisynth to check if it's a server or an AVS problem, for example with a GPU benchmark tool. As a fallback KNLMeansCL is a nice alternative as it's also temporal.

@ENunn: Try updating to 3.7.2 test3, some critical bugs were fixed there, just in case to rule out other possibilities.

[kedautinh12]

@TDS: you can try change radius=1 to preset="low" here
https://github.com/Dogway/Avisynth-Scripts/blob/b4b430d9f0f35e5ebaf7546d769f88a5b9fddeed/SMDegrain%20v3.4.7d/SMDegrain%20v3.4.7d.avsi#L248

[guest]

@TDS: you can try change radius=1 to preset="low" here
https://github.com/Dogway/Avisynth-Scripts/blob/b4b430d9f0f35e5ebaf7546d769f88a5b9fddeed/SMDegrain%20v3.4.7d/SMDegrain%20v3.4.7d.avsi#L248

Hi,

Is this what you mean :-

(prefilter== 6) ? pref.ex_BM3D(sigma=10,preset="low",CUDA=true,UV=Chr,tv_range=tv)

[kedautinh12]

Yes, if you want faster, try preset="fast"

[guest]

Yes, if you want faster, try preset="fast"

Thnx, initially I thought it had fixed it, but it hasn't :(

I still keep getting this error, on the older PC's:-

x265 [error]: could not open input file `-'

[guest]

@TDS: I think your issue would probably be one of resources distribution through the server. Maybe CUDA instructions are not passed, you can try first outside avisynth to check if it's a server or an AVS problem, for example with a GPU benchmark tool. As a fallback KNLMeansCL is a nice alternative as it's also temporal.

@ENunn: Try updating to 3.7.2 test3, some critical bugs were fixed there, just in case to rule out other possibilities.

Curious that you're suggesting KNLMeansCL, and just a short time ago, you removed it....hmmmm. :scared:

[kedautinh12]

Thnx, initially I thought it had fixed it, but it hasn't :(

I still keep getting this error, on the older PC's:-

x265 [error]: could not open input file `-'

The problem from x265 not SMDegrain

[WolframRhodium]

So my BM3D woe's continue.

I have several older PC's that aren't AVX2, BUT they have reasonable nVidia GPU's for CUDA / OpenCL support.

Naturally BM3D CPU doesn't work, BUT I just can't get BM3D CUDA to work on them. I really thought I got it to work before...but can't get it to, anymore :(

KNLMeansCL & DGDenoise work fine.

So do I just forget about using BM3D pre-filters ???, what's the next best thing ??

I'm tearing my hair out (and I haven't got much of that anyway) !!

Help...

Is there any useful error message? What GPU and driver are you using?

[guest]

The problem from x265 not SMDegrain

Maybe so, but how come it works on some PC's, and not others ???

That's the biggest problem, it's random, or should I say, particular. :(

And I also get this when using x264 :-

y4m [error]: bad sequence header magic
x264 [error]: could not open input file `-'

[guest]

Is there any useful error message? What GPU and driver are you using?

Hello,

The only errors I see displayed are in the previous post(s), (842 & 846)...that's the ONLY indication I get with the app I'm using, which no one here uses or know's much about.

I have several PC's with nVidia GPU's ranging from a GTX 660, 760, 770, 780ti, and a 980ti hybrid, I used to have an MSI 1080 Gamer X, but it developed errors, and is basically un usable.

They are all (except the 980ti) using the latest available driver, 472.12.

Maybe I need to install the CUDA development tools or something.

Now several of these PC's have older Xeon's that don't support AVX2, and that's why I need CUDA to work, other wise I will have to use other non Cuda filters...if only you could take the time to mod BM3D avs, to do that :)

Anyway, that's about all I can tell you.

[Boulder]

You have to open the script for example in VirtualDub, which will tell you the error message.

[guest]

You have to open the script for example in VirtualDub, which will tell you the error message.

Yes, I tried that when you suggested it a few days ago..

Is virtualdub2 the same ??

Anyway, if I try to open any .avs, it fails.

I think I might have figured out how to open avs, it has to be a video file..

Anyway, the scripts I tested were ALL faulty, so that's not right.

[Boulder]

Yes, I tried that when you suggested it a few days ago..

Is virtualdub2 the same ??

Anyway, if I try to open any .avs, it fails.

VDub2 is the recommended one. Make sure you use the correct version (32-bit or 64-bit) depending on your system and Avisynth setup. If it fails, it should give you a meaningful error message as well. We cannot see what you see unless you tell us.

[guest]

VDub2 is the recommended one. Make sure you use the correct version (32-bit or 64-bit) depending on your system and Avisynth setup. If it fails, it should give you a meaningful error message as well. We cannot see what you see unless you tell us.

You must have replied to this one, before I edited my post...

Yes, I opened a few scripts, and even the ones that work for me threw up a error message pop up.

I used VDub264, as I'm using Windows 10 x64 & Avisynth 3.7.1.

I would have to take a screen shot of the error message (or whatever it is).

One thing I did notice, it seemed to reference mvtools2 in most msg's.

[kedautinh12]

Maybe your CPU Xeon error with x265 and x264

[guest]

Maybe your CPU Xeon error with x265 and x264

I doubt it would happen on all of them.

I tried a different build of x264 & x265...same thing.

But it only seems to be BM3D...Knlmeans & the new DG filters work.

I know that BM3DCPU won't work due to no AVX2, but BM3DCUDA should !!!

[StainlessS]

One thing I did notice, it seemed to reference mvtools2 in most msg's.
Very mystical, so you want everybody to just guess what the problem might be.
Post the error message. [just write down the text, no screenshot]

[guest]

Very mystical, so you want everybody to just guess what the problem might be.
Post the error message. [just write down the text, no screenshot]

OK StainlessS, see what you make of this :-

Avisynth open failure:
Cannot load file 'C:/RipBot264/Tools/Avisynth plugins/Scripts/Custom/%AVISYNTHPLUGINS%/mvtools/mvtools2.dll'.
Platform returned code 126:
The specified module could not be found.

(C:\RipBot264\Tools\Avisynth plugins\Scripts\Custom\TemporalDegrain.avs,line 2)


Avisynth open failure:
Cannot load file 'C:/RipBot264/Tools/Avisynth plugins/Scripts/Custom/%AVISYNTHPLUGINS%/mvtools/mvtools2.dll'.
Platform returned code 126:
The specified module could not be found.

(C:\RipBot264\Tools\Avisynth plugins\Scripts\Custom\SMDegrain E-H BM3D CPU.avs,line 2)

https://imgur.com/23BJTtn

Why are some like this /, and others like this \....

Now both of these scripts work (on some of my PC's)

So I don't know why they have errors.

It does suggest that maybe there's something wrong with mvtools2.dll, but what ??

[StainlessS]

Platform returned code 126:
You are missing some requirement/runtime.

Here is the AIO (all in one) CPP runtime setup:- https://github.com/abbodi1406/vcredist/releases
Updated 23 days ago [has all cpp runtimes, smaller, less crap than M$ collection of runtimes, both x86 and x64 - recomended].
Above, probably the problem.

EDIT: "\" and "/" just depends what issues the error message, one is DOS style slash, other is unix style slash.

[guest]

You are missing some requirement/runtime.

Here is the AIO (all in one) CPP runtime setup:- https://github.com/abbodi1406/vcredist/releases
Updated 23 days ago [has all cpp runtimes, smaller, less crap than M$ collection of runtimes, both x86 and x64 - recomended].
Above, probably the problem.

EDIT: "" and "/" just depends what issues the error message, one is DOS style slash, other is unix style slash.

Hi, I only updated this a couple of days ago...I make sure I keep up to date on most things.

Got from MS, tho.

I will give this a shot, and report back, probably tomorrow, now.

Cheers & thanks.

[StainlessS]

I think that Mvtools also requires either libfftw3f-3.dll or fftw3.dll for certain modes.
EDIT: https://github.com/pinterf/mvtools
External dependencies:

libfftw3f-3.dll (or renamed to FFT3W.DLL) from http://www.fftw.org/ or look at ICL builds at http://forum.doom9.org/showthread.php?t=173229 It is used only at specific dct parameter values

May require Microsoft Visual C++ Redistributables https://www.visualstudio.com/downloads/

[guest]

I think that Mvtools also requires either libfftw3f-3.dll or fftw3.dll for certain modes.
EDIT: https://github.com/pinterf/mvtools

Yes, I'm pretty sure I have all that installed, but of course I will double check.

Cheers

[kedautinh12]

OK StainlessS, see what you make of this :-



https://imgur.com/23BJTtn

Why are some like this /, and others like this \....

Now both of these scripts work (on some of my PC's)

So I don't know why they have errors.

It does suggest that maybe there's something wrong with mvtools2.dll, but what ??

From your log, i seen this errors from mvtool2 not bm3dcuda

[guest]

From your log, i seen this errors from mvtool2 not bm3dcuda

Exactly, but if there was really a problem with it, I'm sure it would affect other filters & scripts.

Anyway, StainlessS has suggested a few things as well, so I will see how I go next time.

Cheers

[guest]

I think that Mvtools also requires either libfftw3f-3.dll or fftw3.dll for certain modes.
EDIT: https://github.com/pinterf/mvtools

OK, installed that CPP runtime setup package, and got a fresh mvtools2, and VD2 throws up the exact same error....

[StainlessS]

Well if you are using Avisynth x64, you need [from Dependency Walker]

x64 Visual C ++ Redistributable for Visual Studio 2015 package.
MSVCP140.dll
VCRuntime140.dll
VCRuntime140_1.dll

In your system32 [installed with x64 VS 2015 redist, or AIO package]

Also need x64 versions
libfftw3f-3.dll (or renamed to FFT3W.DLL) [some plugs require one name, some the other]
in system32

Other than that, I know of no other dependencies.

[WolframRhodium]

I have several PC's with nVidia GPU's ranging from a GTX 660, 760, 770, 780ti, and a 980ti hybrid, I used to have an MSI 1080 Gamer X, but it developed errors, and is basically un usable.

Thanks. The test6 build (https://github.com/WolframRhodium/VapourSynth-BM3DCUDA/files/8061593/BM3DCUDA_AVS-test6.zip) should work on 780ti and 980ti. Other gpus are too old to be supported.

980ti still requires driver >=452.39.

[tormento]

test6 build
Thanks! I will try it ASAP.

Did you statically linked VC libraries? The dlls seem to me a bit huge in size. Is it possible to have the dynamic version too?

[tormento]

Other gpus are too old to be supported.

I did some benchmarks with SMDegrain on test 5 and test 6 with a 1080p video and x264 --preset slow.

SetMemoryMax()
SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\2_11 8,1 Departures\departures.dgi",ct=24,cb=24,cl=0,cr=0)
ConvertBits(16)
SMDegrain (tr=3, thSAD=300, refinemotion=true, contrasharp=false, PreFilter=6, plane=4, chroma=true)
fmtc_bitdepth (bits=8,dmode=8)
Prefetch(6)

test 5 5.91 fps

test 6 6.10 fps

tr=4, thSAD=400

test 5 4,99 fps

test 6 4,71 fps

tr=5, thSAD=500

test 5 4,45 fps

test 6 3,58 fps

Any idea about the decreased performance?

[guest]

Thanks. The test6 build (https://github.com/WolframRhodium/VapourSynth-BM3DCUDA/files/8061593/BM3DCUDA_AVS-test6.zip) should work on 780ti and 980ti. Other gpus are too old to be supported.

980ti still requires driver >=452.39.

I'm puzzled that the other 700 Series GPU's are too old, when they are the same gen & series as the 780 Ti, and use the same driver ?!?!?

Also, there is a new nVidia driver(s) out for these cards.

600 & 700 Series cards for W10 & W11, is 472.98 (Std & DCH)

900 Series W10 & W11, is 472.12 Std or 511.70 DCH

Also, I noticed that you've "remove avx requirement", now as BM3D for Avisynth didn't support AVX before, how can it be removed, UNLESS you mean you've added AVX support.

Anyway, I will be testing new build(s) soon.

[Dogway]

@tormento: Try with ex_BM3D() only without other filters, and do like 4 or 5 runs for each to calculate the average to get reliable numbers.

[guest]

Has SMDegrain v3.4.7d been changed back to BM3DCUDA by default ??

[WolframRhodium]

I'm puzzled that the other 700 Series GPU's are too old, when they are the same gen & series as the 780 Ti, and use the same driver ?!?!?

Their compute capabilities (https://developer.nvidia.com/cuda-gpus) (a term used by NV to denote the availability of hardware features) are different. CUDA 11 requires cc >= 3.5 (and deprecates 3.5 <= cc <= 5.0, which may be dropped in CUDA 12).


Also, I noticed that you've "remove avx requirement", now as BM3D for Avisynth didn't support AVX before, how can it be removed, UNLESS you mean you've added AVX support.

Anyway, I will be testing new build(s) soon.

AVX (not AVX2) is required in the previous avs+ builds. Now the requirement is restored in test7 (https://github.com/WolframRhodium/VapourSynth-BM3DCUDA/files/8065185/BM3DCUDA_AVS-test7.zip). test6 is accidentally build with debug config so it should not be used for production, sorry for the mistake.

[guest]

Their compute capability (https://developer.nvidia.com/cuda-gpus) (a term used by NV to denote the availability of hardware features) is different. CUDA 11 requires cc >= 3.5 (and deprecates 3.5 <= cc <= 5.0).



AVX (not AVX2) is required in the previous avs+ builds. Now the requirement is restored in test7 (https://github.com/WolframRhodium/VapourSynth-BM3DCUDA/files/8065185/BM3DCUDA_AVS-test7.zip). test6 is accidentally build with debug config so it should not be used for production, sorry for the mistake.

Hello again, well, I have some news for you then.

I got the latest nVidia drivers for the 600 & 700 series, and your "test#6", and also your #35 & #36 releases on Git.

Now Dogway has changed things around again, I think, yet to deal with his latest versions, anyway I tested #6, still the same, then I tried #35, and it worked on the GTX680 SOC (I made a typo yesterday, I haven't got a 660)...then I tried #36, and it worked too.

I will try "test#7", and let you know.

Cheers

"Test #7" doesn't work for me, but it doesn't matter if the others do...

I just noticed that you posted some more builds on Git, #38 & #42, so I got them, and neither work for me, in fact they didn't even work on the 980Ti PC, where ALL the others do.

[Boulder]

Do you have any recommended deblocking function from your package regarding crappy VC1 sources? I'm currently working on the Star Trek TOS blu-rays, and the quality is far from good. I was thinking that some kind of deblocking (+ possibly adding noise to deceive the brain) might be useful here. Avoiding any serious detail loss would be a big plus.

[kedautinh12]

I think sample is needed. I prefer SmoothD2c, oyster and autodeblock

[Dogway]

I rarely denoise spatially, so usually I apply deblocking as prefilter (a strong deblock is fine) and let SMDegrain clean up the rest. Where block matching fails you are gonna get blocking so use also a deblocker for 'mfilter'.

[Boulder]

Here are two sample clips, the first one shows the blocking in the flat background areas. To my eyes, it looks rather sharp, maybe it's something typical of VC1. The second clip has some closeups and darker areas. It doesn't look as bad because the blocking is not as apparent in motion.

https://drive.google.com/file/d/15PFYjZB2JgFMtGydRr8eA2vg1TxoqyxB/view?usp=sharing
https://drive.google.com/file/d/1ws2B4t9ZiaO1e-LUwBH8Vi2-ntWRQwQf/view?usp=sharing

[tormento]

Try with ex_BM3D() only without other filters, and do like 4 or 5 runs for each to calculate the average to get reliable numbers.
Yes, I always average on 5 runs.

I did a x264 comparison test because, at the end of the day, it's the global performance what matters mostly. I know there could be some minor fluctuations due to video compressibility but on the very same file, with the very same script, unless a big plugin engine modification, they are negligible.

To complete the table:

SetMemoryMax()
SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\2_11 8,1 Departures\departures.dgi",ct=24,cb=24,cl=0,cr=0)
ConvertBits(16)
SMDegrain (tr=3, thSAD=300, refinemotion=true, contrasharp=false, PreFilter=6, plane=4, chroma=true)
fmtc_bitdepth (bits=8,dmode=8)
Prefetch(6)

test 5 5.91 fps
test 6 6.10 fps
test 7 6,58 fps
test 8 6,03 fps

tr=4, thSAD=400

test 5 4,99 fps
test 6 4,71 fps
test 7 5,01 fps
test 8 5,56 fps

tr=5, thSAD=500

test 5 4,45 fps
test 6 3,58 fps
test 7 4,29 fps
test 8 3,64 fps

I did, anyway, some tests with AVSMeter on plain ex_BM3D() too.

SetMemoryMax()
SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\2_11 8,1 Departures\departures.dgi",ct=24,cb=24,cl=0,cr=0)
ex_BM3D()
Prefetch(6)

test 5 18,58 fps
test 6 16,55 fps
test 7 19,47 fps
test 8 19,56 fps

But now comes the funny part. If I modify the script as:

SetMemoryMax()
SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\2_11 8,1 Departures\departures.dgi",ct=24,cb=24,cl=0,cr=0)
ConvertBits(16)
ex_BM3D()
fmtc_bitdepth (bits=8,dmode=8)
Prefetch(6)

test 8 20,86 fps

and with

fmtc_bitdepth (bits=10,dmode=8)

test 8 20,45 fps

P.S: I'd like really to try a version of your scripts where ConvertBits is replaced by fmtc_bitdepth, as I find it faster.

[Boulder]

I rarely denoise spatially, so usually I apply deblocking as prefilter (a strong deblock is fine) and let SMDegrain clean up the rest. Where block matching fails you are gonna get blocking so use also a deblocker for 'mfilter'.

I started experimenting with SmoothD2c (since I've already used SmoothD2 back in the day), line 988 in deblockpack.avsi throws an error that 'fs' is unknown.

zw1 = (Cpr == 0) ? sd1.ex_Ylevels( 0, 1.0, 255, 0, 0, fs) : sd1.buildZmask(cpr=cpr)

After setting false there, the next error is from line 999: SmoothD2: input to filter must be in YV12. This is after loading the source in DGSource, so it's regular 8-bit YV12 data.

[gispos]

Hello Dogway,
I don't know if you will read my post in the confusion of postings here, but I'll throw it in with the other postings.:)

As a note on the AvsPmod slider function definetions:
All parameters listed in the function header must also be present in the slider templates.
Otherwise if the user sets a parameter that is not in the definition, the slider window remains empty (the function is not recognized).

For parameters that should not be displayed or for which no default value should be given, the parameter must be enclosed in a square bracket.
As an example ex_contrast and "UV".

(
clip,
float "cont"=0.00 (-1.00 to 1.00),
float "pivot"=127.5 (12.0 to 192.0 by 0.5),
float "sat"=1.00 (0.00 to 3.00),
[int "UV"],
bool "tv_range"=True,
bool "show"=False
)

[Dogway]

@Boulder, AFK today, I will check SmoothD2c tomorrow, I'm not sure if it still was in WIP or not.

@gispos, very nice I didn't know the trick to omit defaults, as for brackets I use them occasionally but I don't always include all args into sliders as to not make them "busy".

Man, little things here and there and still haven't resumed the SceneStats() filter. I plan to do a few "minor" updates tomorrow and probably resume it.

[Boulder]

Also an issue with GrainFactory3mod.avsi: if you set the tv_range parameter, an error "I don't know what 'tv' means" is shown.

[Boulder]

One more possible issue that I found: at least QTGMC(preset="slower", noisepreset="fast", tr2=0, inputtype=2, noiseprocess=2, grainrestore=0.5, noiserestore=0.3) changes the _ColorRange property from 1 to 0.

While looking at the props, I was thinking that should QTGMC set the _FieldBased property to 0?

[kedautinh12]

I think you need creat issues here
https://github.com/Dogway/Avisynth-Scripts/issues

[Dogway]

Thanks Boulder, just fixed GrainFactory3mod and QTGMC.
EDIT: Also fixed SmoothD2c()

[guest]

Hey Dogway,

I just noticed the update you've done with DeblockPack.avsi (that I don't use)...

Could you please provide an appropriate script or call to use this ??

Sounds interesting with the BM3D "connection".

Thanks

[Dogway]

You mean Oyster? it's feisty2's deblocker/deringing/AA, it's very slow and currently in WIP. I need to dedicate it more time. Read more here (https://forum.doom9.org/showthread.php?t=173470).

@tormento: I benchmarked fmtc_bitdepth() and resulted slower than internal, even using dmode=2.

206.9fps
fmtc_bitdepth(bits=16)
removegrain(12)
fmtc_bitdepth(bits=8,dmode=2)
fmtc_bitdepth(bits=16)
removegrain(12)
fmtc_bitdepth(bits=8,dmode=2)

208.4fps
ConvertBits(16)
removegrain(12)
ConvertBits(8,dither=-1)
ConvertBits(16)
removegrain(12)
ConvertBits(8,dither=-1)

[Boulder]

Got one more changed prop: GrainFactory3mod (at least the MIX version) switches from limited to full range.

[tormento]

@tormento: I benchmarked fmtc_bitdepth() and resulted slower than internal, even using dmode=2.
Geez... It's less than 1%... Won't we have benefits using better dithering modes that fmtc_bitdepth provides?

[Dogway]

Thanks Boulder, fixed already.

Geez... It's less than 1%... Won't we have benefits using better dithering modes that fmtc_bitdepth provides?

Maybe, but all internal down conversions to 8-bit in my scripts use dither=-1 so that's the most used. Other than that is only applied once at the script end typically. I don't think it doesn't make much difference speed wise, in quality though I do think there are benefits in using other than floyd-steinberg but haven't reached to conclusions. One simple way to check dither types is with ex_posterize().

[Zetti]

how to fix this error?

https://i.ibb.co/Q9ktf07/Screenshot-2022-02-17-160108.png

[Boulder]

how to fix this error?

https://i.ibb.co/Q9ktf07/Screenshot-2022-02-17-160108.png

Open that file in Notepad and see how it looks. I haven't had any issues with the latest ExTools.avsi so chances are there's something wrong with the format.

[Zetti]

It looks like a normal script. I can't see anything wrong (it's the first time i have looked in ExTools.avsi)

[Shinkiro]

I can't figure out what I'm not doing wrong, seem all the clips YUV420P16
mthr = 34

ConvertBits(16)
Source=last
mask1b=last.ex_edge("kirsch",6,10)
mask2b=last.ex_edge("kirsch",4,7)
mask=last.ConditionalFilter(mask1b, mask2b, "AverageLuma()",">","50").mt_lut("x "+string(mthr)+" <= x 1 >> x 1 << ?", U=1, V=1).RemoveGrain((980>960) ? 20 : 11, -1)

deb = Source.neo_f3kdb(sample_mode=2, Y=68, Cb=68, Cr=68, grainy=54, grainC=40, range=15, dynamic_grain=true)
ex_merge(deb ,Source ,mask, luma=true, Y=3, UV=3)
returns an error
CombinePlanes: source and target plane dimensions are different
(C:/AviSynth+/plugins64+/ExTools.avsi, line 7072)
(C:/AviSynth+/plugins64+/ExTools.avsi, line 527)

why does he say that it is not so, or does ex_merge not know how to handle 10-16bit?

[Boulder]

It looks like a normal script. I can't see anything wrong (it's the first time i have looked in ExTools.avsi)

In my version (the latest one), the line looks like this and in column 47, there's no [ but A.

z4 = mcfa || mcf == "rMEA" ? Expr(yA[0],yA[1],yA[2],bA[0],bA[1],bA[2],"x a + 0.5 * R@ 0.00390625 * 2 + x a - dup * * y b - dup * 4 * + 255 R - 0.00390625 * 2 + z c - dup * * + sqrt") : nop()

[Zetti]

It looks also like this in the version I have downloaded today.

[Dogway]

returns an error

why does he say that it is not so, or does ex_merge not know how to handle 10-16bit?

Shinkiro, what's the size of your source clip? I think I know what the issue might be, a size where chroma is odd size, but just to be sure.

how to fix this error?

https://i.ibb.co/Q9ktf07/Screenshot-2022-02-17-160108.png

What AVS+ version are you, and AvsPmod version? Also show your script, I haven't run into such errors.

[Shinkiro]

Source BDMV

ID/String : 0 (0x0)
Format/String : BDAV
Format/Info : Blu-ray Video
FileSize/String : 8.76 GiB
Duration/String : 29 min 27 s
OverallBitRate_Mode/String : Variable
OverallBitRate/String : 42.6 Mb/s
OverallBitRate_Maximum/String : 48.0 Mb/s

Video

ID/String : 4113 (0x1011)
MenuID/String : 1 (0x1)
Format/String : AVC
Format/Info : Advanced Video Codec
Format_Profile : High@L4.1
Format_Settings : CABAC / 4 Ref Frames
Format_Settings_CABAC/String : Yes
Format_Settings_RefFrames/String: 4 frames
CodecID : 27
Duration/String : 29 min 27 s
BitRate_Mode/String : Variable
BitRate/String : 38.0 Mb/s
Width/String : 1 920 pixels
Height/String : 1 080 pixels
DisplayAspectRatio/String : 16:9
FrameRate/String : 23.976 (24000/1001) FPS
ColorSpace : YUV
ChromaSubsampling/String : 4:2:0
BitDepth/String : 8 bits
ScanType/String : Progressive
Bits-(Pixel*Frame) : 0.764
StreamSize/String : 7.78 GiB (89%)

[Dogway]

Standard 1080p. It works on my end though I can't see the issue.
What AVS+ version are you on?

Please try with the next and tell me if you still get the error:

ConvertBits(16)
Source=last
mask1b=last.ex_edge("kirsch",6,10)
mask2b=last.ex_edge("kirsch",4,7)
mask=last.ConditionalFilter(mask1b, mask2b, "AverageLuma()",">","50").mt_lut("x "+string(mthr)+" <= x 1 >> x 1 << ?", U=1, V=1).RemoveGrain((980>960) ? 20 : 11, -1)

mskY_to_YYY(last,mask,true,false,3,16)

I only got that error if I tried to use the above over an RGB clip.

[Shinkiro]

Please try with the next and tell me if you still get the error:


with this script I get the same error

CombinePlanes: source and target plane dimensions are different
(C:/AviSynth+/plugins64+/ExTools.avsi, line 7072)

I tried it on another video, the result is the same
Avisynth_3.7.2_test3, ExTools v7.5

[Zetti]

What AVS+ version are you, and AvsPmod version? Also show your script, I haven't run into such errors.

1. Avisynth+ 3.7.2 test 3

2. I do not use AvsPmod

3. I have no script yet because i getting the error when MeGUI has created index file, and trying to load the index file in "MeGUI's Avisynth script creator"

4. FFMS indexer giving me that error, but DGIndexNV indexer giving me this error

https://i.ibb.co/HK506Vq/Screenshot-2022-02-18-084553.png

If i moving your scripts out of the plugin folder, there is no problem when creating index file and basic script i MeGUI

Normally when i download updated scripts and plugins, i just replace the files in the plugin folder and im not getting any errors that i cannot fixing myself

But this time i can't find out what causing these errors

I just want to use SMDegrain

[Dogway]

Maybe MeGUI is parsing the scripts and not understanding the array syntax.
What I can ask you to do is to test your script in AVSmeter so I can know it's not a conflict with my scripts but with MeGUI.

Simply run:
"D:\Rip\workshop\AVSMeter64.exe" "Path/of/your/script.avs"

[Zetti]

MeGUI seems to be the problem.

Becuase i moved your scripts out of the plugin folder and created my basic script with only the index file, and i have moved your scripts back.

Now i have tested my basic script and it's working with AVSMeter.

With your scripts in the plugin folder, i couldnt create the basic script with MeGUI.

So from now on, i need to import all the required plugins and scripts from another folder into my encoding script.

[Dogway]

Well thanks for testing, I checked and in "MeGUI/tools/avs" folder you find the avisynth dll files, you can try replacing those with v3.7.2 test3, just an idea.

[kedautinh12]

You can try L-SMASH in megui. And i use extools in megui normally

[Zetti]

Maybe i should check the Avisynth and AvisynthWrapper versions one more time.

And testing L-SMASH.

If that not getting it to work, a seperate folder could be the solotion.

[kedautinh12]

If you use latest avs+ ver, you need change new AvisynthWrapper to use latest avs+ ver
https://forum.doom9.org/showthread.php?t=181351

[Zetti]

I got it working without any errors after replacing the old versions of Avisynth and AvisynthWrapper with new ones in MeGUI.

Can you both have a great and wonderfull day.

[Boulder]

There seems to be some sort of a memory leak in QTGMC, at least when SRestore is involved. This script causes the memory usage to creep up until it hits the set memory limit, the Avisynth basically stalls. It does proceed, but at a very, very slow pace.
Replacing QTGMC with real.finder's STGMC fixes the issue, the memory usage stays below 1GB all the time.

SetMemoryMax(8192)
DGSource("source.dgi") # 1080i, 29.97 fps
QTGMC(preset="medium", noisepreset="fast", tr2=0, sigma=0.25)
Spline36Resize(1280,720)
SRestore()

[Dogway]

I'm pretty sure it's a memory leak in pixel addressing, it has been reported almost 2 months ago (https://forum.doom9.org/showthread.php?p=1960946#post1960946), and recalled 2 weeks ago (https://forum.doom9.org/showthread.php?p=1963618#post1963618) but pinterf has been turning a blind eye on it.

I will try to cobble up a version without pixel addressing so you can confirm. It's either that or a LUT bug.

[real.finder]

I'm pretty sure it's a memory leak in pixel addressing, it has been reported almost 2 months ago (https://forum.doom9.org/showthread.php?p=1960946#post1960946), and recalled 2 weeks ago (https://forum.doom9.org/showthread.php?p=1963618#post1963618) but pinterf has been turning a blind eye on it.

I will try to cobble up a version without pixel addressing so you can confirm. It's either that or a LUT bug.

maybe it's better to post it here https://github.com/AviSynth/AviSynthPlus/issues

[Dogway]

@Boulder: Test with this (https://pastebin.com/tmscgheP). I removed all pixel addressing calls, try to use the same threads than in STGMC, I don't know what's the default there.

[Boulder]

@Boulder: Test with this (https://pastebin.com/tmscgheP). I removed all pixel addressing calls, try to use the same threads than in STGMC, I don't know what's the default there.
There's a function missing:
error: Script error: There is no function named 'QTGMC_deflate'.
(C:/Program Files/AviSynth+/plugins64/qtgmc.avsi, line 1064)
(C:/Program Files/AviSynth+/plugins64/qtgmc.avsi, line 907)

[Boulder]

The issue might be partly related to SRestore as well. Running it in HBD also causes the leak with STGMC. It would be interesting to see if the tuned version of QTGMC handles 8-bits though, since the original version didn't. I'll doublecheck that one more time though.

SetMemoryMax(8192)
DGSource("source.dgi") # 1080i, 29.97 fps
STGMC(preset="medium", noisepreset="fast", tr2=0, sigma=0.25)
ConvertBits(16)
Spline36Resize(1280,720)
SRestore()

[Dogway]

Yes sorry, here are both functions from old QTGMC:


# Functions (from original TGMC) used instead of mt_xxflate with similar operation but a somewhat stronger result. Originally added for speed, they are
# no longer faster due to improvements in masktools. Difference (visual and speed) is small so may be reverted in a later version.
function QTGMC_inflate( clip c, int "Y", int "U", int "V" )
{
# >>>> YUY2 is planar here
mtY =default( Y, 3 )
mtU =default( U, 1 )
mtV =default( V, 1 )
rgY = (mtY == 3) ? 20 : -1
rgU = (mtU == 3) ? 20 : -1
rgV = (mtV == 3) ? 20 : -1
mt_logic( c, c.RemoveGrain( rgY, rgU, rgV ), "max", Y=mtY,U=mtU,V=mtV )
}

function QTGMC_deflate( clip c, int "Y", int "U", int "V" )
{
# >>>> YUY2 is planar here
mtY =default( Y, 3 )
mtU =default( U, 1 )
mtV =default( V, 1 )
rgY = (mtY == 3) ? 20 : -1
rgU = (mtU == 3) ? 20 : -1
rgV = (mtV == 3) ? 20 : -1
mt_logic( c, c.RemoveGrain( rgY, rgU, rgV ), "min", Y=mtY,U=mtU,V=mtV )
}

If STGMC also shows a memory leak in HBD (16-bit I understand) it could be a LUT problem. LUT calculations, at least for 16-bit are enabled for ex_lut()/mt_lut() only.

One more thing, it has been a long time I used srestore() but as far as I recall it was very picky with frame access. I used to use it only with MP_Pipeline() for these reasons or at another pass. You can also try to put RequestLinear() before the call.

[Dogway]

On another note, I coded a few new filters for ExTools, usually this involves several iterations that don't make the cut to the final version but there still might be some value on them.
This time it happened with Harris corner detection, I created one version that differs from the canonical one from the papers but IMO performs better.
So here it is the "by the book" Harris corner detection with helper function Non-Maximum Suppression.

# Harris corner detection
#
# Alternative version which works well for small images and graphics
function ex_corner_canon(clip a, float "sigma", float "th", bool "show") {

rgb = isRGB(a)
isy = isy(a)
bi = BitsPerComponent(a)

rd = Default(sigma, 1.4)
th = Default(th, 0.7) # From 0.1 to 0.9
sh = Default(show, false)
th = th/10000.

a = ConvertBits(a,32)
a

x1 = ex_lut("x[0,-1] x[0,1] - dup *").ex_blur(rd,mode="gaussian")
y1 = ex_lut("x[-1,0] x[1,0] - dup *").ex_blur(rd,mode="gaussian")
xy = ex_lutxy(x1,y1,"x y * ").ex_blur(rd,mode="gaussian")
# xy = ex_lut("x[0,-1] x[0,1] - abs x[-1,0] x[1,0] - abs max").ex_blur(rd,mode="gaussian")

ex_lutxyz(x1,y1,xy,"x y * z - x y + dup * 0.05 * -",UV=2) # R = Det(M) - k*Trace(M)^2
ex_NMS(thin=1.0) # In the paper this goes after thresholding, but this resulted better on my tests
ex_lut(Format("x {th} > range_max 0 ?"),UV=2)
# Alternative without suppression
# ex_lutxyz(x1,y1,xy,Format("x y * z - x y + dup * 0.04 * - {th} > range_max 0 ?"),UV=2)

# Corner augment
ex_expand(3,mode="disk").ex_deflate()

# Show
sh ? ex_blend(a,last, "linearadd", 0.4, 1.0) : last
ConvertBits(bi) }


# Non-Maximum Suppression (NMS) - Edge Thinning Algorithm
#
# ex_NMS(thin=0.3) # 15.5 (164 without scaling)
# awarpsharp2(chroma=2, depth=12, blur=4, thresh=128) # 167
# aWarpSharp(depth=8,blurlevel=2,thresh=0.5) # 182
#
function ex_NMS(clip a, float "thin", bool "show") {

ss = Default(thin, 0.5) # Super sampling factor
sh = Default(show, false)
ss = min(4.0,1./ss)

RatioResize(a,ss,"%",kernel="blackman",src_left=0.25,src_top=0)

ex_lut("x[0,-1] T@ x[0,1] B@ -
x[-1,0] L@ x[1,0] R@ -
atan2 0.1591549431 * dup 0 > swap dup 1 + ? 180 * A@

157.5 > x[0,0] X@ T > X B > & X L ?
A 112.5 > X x[1,-1] UR@ > X x[-1,1] DL@ > & X DL ?
A 67.5 > X L > X R > & X T ?
X x[-1,-1] UL@ > X x[1,1] > & X UL ? ? ? ? ",UV=2)

RatioResize(1./ss,"%",kernel="bicubic",b=0,c=1)

ex_limitdif(a, last, thr=2, elast=5, UV=2) }

[Boulder]

I think I found the culprit after many rounds of debugging - it's propCopy and that's why your QTGMC leaks memory since STGMC doesn't use the props feature at all.

This won't stall:
DGSource("source.dgi")
STGMC(preset="medium", noisepreset="fast", tr2=0, sigma=0.25)
propSet("_FieldBased", 0)
Spline36Resize(1280,720)
SRestore()

While this will:
DGSource("source.dgi")
org=last
STGMC(preset="medium", noisepreset="fast", tr2=0, sigma=0.25)
propCopy(last, org)
propSet("_FieldBased", 0)
Spline36Resize(1280,720)
SRestore()

I think I need to report this in GitHub along with the other property related issue I found earlier.

[Dogway]

Oh, that one was also reported (https://forum.doom9.org/showthread.php?p=1956036#post1956036) back in October, still no word from pinterf.

So you are saying that my QTGMC mod without propCopy is fine? Or do you want to keep debugging it?

[guest]

Can someone explain why this is happening...

I have a PC with an old AMD R9 290 GPU (which DOES NOT support CUDA), and when I use BM3D CUDA, of course it fails, BUT, I have also installed a basic GTX 650 (which DOES support CUDA).

And now, if I use BM3DCUDA, both are encoding, using OpenCL.

The R9 290 is a LOT faster as it has 40 compute units, whereas the poor little GTX 650 only has 2.

[poisondeathray]

Can someone explain why this is happening...

I have a PC with an old AMD R9 290 GPU (which DOES NOT support CUDA), and when I use BM3D CUDA, of course it fails, BUT, I have also installed a basic GTX 650 (which DOES support CUDA).

And now, if I use BM3DCUDA, both are encoding, using OpenCL.

The R9 290 is a LOT faster as it has 40 compute units, whereas the poor little GTX 650 only has 2.

GTX 650 is too old

https://github.com/WolframRhodium/VapourSynth-BM3DCUDA

Requirements

CUDA-enabled GPU(s) of compute capability 5.0 or higher (Maxwell+).

[guest]

GTX 650 is too old

https://github.com/WolframRhodium/VapourSynth-BM3DCUDA

I don't care about requirements, it just WORKS !!!!!!

[poisondeathray]

I don't care about requirements, it just WORKS !!!!!!

ahh I misunderstood;

Maybe CPU fallback ? What were the CPU's between the 2 systems?

How are you measuring OpenCL usage ? What encoder are you using ? (is it using OpenCL as well) ?

[kedautinh12]

GTX 650 is too old

https://github.com/WolframRhodium/VapourSynth-BM3DCUDA

Test 8 now support compute capability 3.5 and higher

[guest]

ahh I misunderstood;

Maybe CPU fallback ? What were the CPU's between the 2 systems?

How are you measuring OpenCL usage ? What encoder are you using ? (is it using OpenCL as well) ?

Well, my post was asking how can this be working ??

It's an i7 3930K. which only supports AVX, so BM3DCPU doesn't work, and when I only had the AMD R9 290 installed, it could not use BM3DCUDA, but now that I've installed the "old" GTX 650, the encoding app seems to be able to use both GPU's when using BM3DCUDA.

The encoding app is RipBot264, which is sadly way out of date, but still works quite well.

I have a Ryzen 3950X with an AMD RX 480 (which doesn't support CUDA), so I put an old GTX 760, and the same applies, it just works :)

I also have a couple of Xeon E5 2690's, with old AMD HD6850's & GTX 650's, and it works the same way :)

[guest]

Test 8 now support compute capability 3.5 and higher

Hi ked, I'm not using Vapoursynth, tho...

[Boulder]

Oh, that one was also reported (https://forum.doom9.org/showthread.php?p=1956036#post1956036) back in October, still no word from pinterf.

So you are saying that my QTGMC mod without propCopy is fine? Or do you want to keep debugging it?

Your debug version worked fine once I commented out line 964 which had the propCopy call.

[WolframRhodium]

BM3DCUDA may run on GTX650, but it is not guaranteed by CUDA. It should not be able to run on R9 290, the usage data may be erroneous.

[kedautinh12]

Hi ked, I'm not using Vapoursynth, tho...

I say about avs ver

[guest]

BM3DCUDA may run on GTX650, but it is not guaranteed by CUDA. It should not be able to run on R9 290, the usage data may be erroneous.

Yes, that is why I asked the question in the first place, I was shocked when it appeared that RipBot264 WAS using the OpenCL (with BM3DCUDA).

With the Distributed Encoding feature of RipBot264 you have the option to use a specific GPU to do the encoding.

So when using BM3DCUDA, and have the R9 290 on one server/port, and the GTX 650 on another server/port?, they both appear to be encoding, and at different speeds, due the difference in the Compute Units of both cards.

However without the GTX 650, BM3DCUDA cannot be used.

[guest]

I say about avs ver

It's just that that link sent me to the Vapoursynth version info. AFAIK.

[kedautinh12]

But name ver i said related avs :D
https://github.com/WolframRhodium/VapourSynth-BM3DCUDA/issues/7#issuecomment-886173346

[guest]

But name ver i said related avs :D
https://github.com/WolframRhodium/VapourSynth-BM3DCUDA/issues/7#issuecomment-886173346

Sorry....but this link was different, but made sense. :)

[pinterf]

Oh, that one was also reported (https://forum.doom9.org/showthread.php?p=1956036#post1956036) back in October, still no word from pinterf.

So you are saying that my QTGMC mod without propCopy is fine? Or do you want to keep debugging it?
I think that was a completely different problem (speed). To address your problem propCopy was changed then in order to not copy all the frame content but to use "SubFrame" internally (you as a script writer do not need to know what env->SubFrame is), which is able to keep frame content without actually copying any byte in its planes. The same "subframe" technique is used to make quick SeparateFields, aligned crop, ExtractX, etc.

I suppose this memory leak appeared after this change, because of this subframe-hack which I have to investigate why it doesn't release its memory or clip reference of I don't know what.

[pinterf]

I'm pretty sure it's a memory leak in pixel addressing, it has been reported almost 2 months ago (https://forum.doom9.org/showthread.php?p=1960946#post1960946), and recalled 2 weeks ago (https://forum.doom9.org/showthread.php?p=1963618#post1963618) but pinterf has been turning a blind eye on it.

I will try to cobble up a version without pixel addressing so you can confirm. It's either that or a LUT bug.
Well, probably I have blind eyes but if you'd follow things on git you would realize that I'm not just sitting at home, eating pizza and watching clouds on the sky, plus I've got an official job. In last November I went to a cinema and watched Dune, that was the last time I had time watching _any_ films.

[pinterf]

I think I found the culprit after many rounds of debugging - it's propCopy and that's why your QTGMC leaks memory since STGMC doesn't use the props feature at all.

This won't stall:
DGSource("source.dgi")
STGMC(preset="medium", noisepreset="fast", tr2=0, sigma=0.25)
propSet("_FieldBased", 0)
Spline36Resize(1280,720)
SRestore()

While this will:
DGSource("source.dgi")
org=last
STGMC(preset="medium", noisepreset="fast", tr2=0, sigma=0.25)
propCopy(last, org)
propSet("_FieldBased", 0)
Spline36Resize(1280,720)
SRestore()

I think I need to report this in GitHub along with the other property related issue I found earlier.
Has someone accidentally minimize the script? I omitted SRestore and resize, replaced STGMC (this must be a new thing I'm not aware of) with QTGMC and it is still consuming more and more memory, I wonder which part of it must be kept in order to still reproduce the problem?

[kedautinh12]

Well, probably I have blind eyes but if you'd follow things on git you would realize that I'm not just sitting at home, eating pizza and watching clouds on the sky, plus I've got an official job. In last November I went to a cinema and watched Dune, that was the last time I had time watching _any_ films.

Dune is very good film for me :D

[pinterf]

Not only propCopy(last, org) fails but other two-clip filters as well, replacing propCopy with Stackvertical(last,org) or Merge(last,org) yields the same effect.
I have to remove functions from QTGMC one-by-one until the phenomenon stops occuring, since probably some other function call combination inside QTGMC triggers this leak.

btw:
propCopy now gets/sets everything.
Would it be good have a new parameter to be able to pass an array of property names?
Then only selected properties would only be extracted and copied?

[Dogway]

Your debug version worked fine once I commented out line 964 which had the propCopy call.

Could you test vanilla without propCopy? So we can discard pixel addressing.


I think that was a completely different problem (speed). To address your problem propCopy was changed then in order to not copy all the frame content but to use "SubFrame" internally (you as a script writer do not need to know what env->SubFrame is), which is able to keep frame content without actually copying any byte in its planes. The same "subframe" technique is used to make quick SeparateFields, aligned crop, ExtractX, etc.

Speed issues typically uncover under the hood issues. But I mean you as a mere programmer don't need to know what hysteresis is, nor even maths at all.

Well, probably I have blind eyes but if you'd follow things on git you would realize that I'm not just sitting at home, eating pizza and watching clouds on the sky, plus I've got an official job. In last November I went to a cinema and watched Dune, that was the last time I had time watching _any_ films.

Now we are two (no time even for Dune :(). I do follow things on git, I was just replying to this post (https://forum.doom9.org/showthread.php?p=1962901#post1962901). You made such a fuss about not reporting but I actually do and most of it pass ignored, so I'm not going to annoy anyone that I know is so busy with other tasks as I see on git.

propCopy now gets/sets everything.
Would it be good have a new parameter to be able to pass an array of property names?
Then only selected properties would only be extracted and copied?

Thanks, finally (https://forum.doom9.org/showthread.php?p=1956543#post1956543) it gets implemented. Typically it's only "_ColorRange" what I need to restore, but having also ALL as an option is useful.

QTGMC is a good Expr() testbench because it takes it to its limits. I hope Boulder can answer if vanilla without propCopy() is fine.

[Boulder]

Could you test vanilla without propCopy? So we can discard pixel addressing.

Yes, it works. The issue appears to be a particularly nasty one and reproducable with a very simple script.

https://github.com/AviSynth/AviSynthPlus/issues/270

I'd like to add my 2c to the GitHub reporting issue as an IT guy. I definitely want to bring up things but even if I do, I don't expect them to be handled. It's always up to the maintainer to decide that, I see reporting just a perfect way to keep things in mind. The issues are very easily lost in the long threads. In your case I've made a difference since you frequent the forum a lot these days.

[Dogway]

I'd like to add my 2c to the GitHub reporting issue as an IT guy. I definitely want to bring up things but even if I do, I don't expect them to be handled. It's always up to the maintainer to decide that, I see reporting just a perfect way to keep things in mind. The issues are very easily lost in the long threads. In your case I've made a difference since you frequent the forum a lot these days.

Thanks for the report. It's a matter of time, if you report to GitHub you are supposed to dedicate enough time to debug, time I don't have, and probably pinterf either.
These days I would rather finish the last touches of my scripts because I'm burned out and switch to other things.

[Boulder]

Thanks for the report. It's a matter of time, if you report to GitHub you are supposed to dedicate enough time to debug, time I don't have, and probably pinterf either.
These days I would rather finish the last touches of my scripts because I'm burned out and switch to other things.

I always try to give enough information to be useful, that's also because of my background. At work, I get quite a lot of requests for assistance which lack a lot of information so I have to either guess or ask too many questions to do something about things.

What comes to being burnt out - your scripts already work very well. It's very much possible to take a break and come back to the ideas in the summer or whenever. Of course, if there's some dependency type of bugs found, it's a bit different but I guess the most used functions are in good shape.

[Dogway]

Yes, normally my reports whenever possible are very low level examples with Expr(), but beyond that I don't have time to debug with other conditions, compare old versions, etc.

I have an internal todo list and it's pretty much over. I had Harris corner there for a few months now, and now for ExTools the only thing left is some bugs in ex_makediff() when show=1, and update propCopy() calls to latest (unreleased) AVS+ version (also to other scripts). I will probably promote it to v8.0. It's not perfect but it's where I reach for now.

The only other thing left is MasksPack, I probably can optimize ex_limitdif() further. And finish Soothe() refactor. That's all, besides the Zopti thingy and SceneStats() which is a big but fun thing to do. There are more but for the future I guess.

[Dogway]

Updated ExTools to v7.6. Now all scripts are promoted to AVS+ v3.7.2 due to expected propCopy() refactor, at which time I will rebase all my scripts.
Here is the new filter ex_corner(), based on Harris corner detection but with some own modifications.
https://i.imgur.com/GjHHdaFh.png

[pinterf]

That propCopy issue is a nasty one, it's not because of properties because this pure script similarly exhibits the 'leak'.
ColorBars(pixel_type="YV12",staticframes=false)
org=last
Bob()
Merge(last, org)

Art of skeletonizing of QTGMC :)

The issue is inherited from times shortly after Neanderthals discovered Avisynth+ (I was able to go back to r1825 from 1996). After two days I think it is internal cache related, an otherwise smart cache size auto-optimization was fooled by the scenario of combining an original and a double rate clip. Resolving the problem is in progress, I've got already a non-memory-exhausting version on my test bench but I have a feeling that this workaround in this form would be suboptimal for other cases.

See the issue here (https://github.com/AviSynth/AviSynthPlus/issues/270)

[Dogway]

Only exhibits the issue for double-rate merging? I trimmed the output of Bob() to match length and still showed memory leak.

Maybe this might be related (don't know) to the optimization (https://forum.doom9.org/showthread.php?p=1959004#post1959004) that was present in Merge() (talking out of my a**). Thanks for looking into it.

[pinterf]

What happens when you put SelectOdd(org) instead of org? (I'm not near my PC) or better: Interleave(org,org)

[Dogway]

What happens when you put SelectOdd(org) instead of org? (I'm not near my PC) or better: Interleave(org,org)

Both options are fine.

By the way, do you know if there's such thing as IsRuntime(), so I can disable LUT in this environment, I can do a try catch though but it's not pretty.

function isRunTime(clip a, bool rgb) {
try {
iRT = rgb ? a.AverageR() : a.AverageLuma()
iRT = true
} catch ( error_msg ) {
iRT = false
}
return iRT }

[pinterf]

Both options are fine.

Technically propCopy(last, Interleave(org,org)) is what you need to use. This will pull the proper frame properties. because Nth original frame becomes (2N)th after QTGMC.

Or else - because of frame doubling - the filter before 'org' (in my example ColorBars, or ffms) is receiving the following frame request pattern (actually copied from my log file):

0, 0, 0, 1, 1, 2, 1, 3, 2, 4, 2, 5, 3, 6, 3, 7, 4, 8, 4, ...
44, 22, 45, 23, 56, 23...


Jumping to and fro is not healthy for any source filter (nor for Avisynth+ internal cache algorithm :) )

Using Interleave for this kind of propCopy will not only make the source of copying correctly placed but will speed up the whole thing - at least it won't make it slow down because of out-of-order frame requests. I hope I didn't write much nonsense. I recommend you trying it out.

[Dogway]

But in this case we are pulling frameprops from frame#0. If using offset though frame order is important, same if run in runtime env.

- Allow propGetXXX property getter functions called as normal functions, outside runtime
By default frame property values are read from frame#0 which index can be overridden by the offset parameter

About Bob()+Merge(), I see that the frame request is N and 2N at the same time so the source loader suffers jumping up and down.

Anyway I tested with "output.propCopy(bobbed)" in QTGMC and it worked fine. But I will wait until you add selective frameprop copy if you still plan to add it.


EDIT: Boulder, updated QTGMC() give it a try now, so far looks fine here. I had to port sh_bob() to inherit some frame properties.

[Boulder]

The new version works fine, no memory leaks whatsoever :)

[salvo00786]

Hi...
Dogway, please can you help me?
I'm trying to use your latest version of QTGMC with Ripbot, but when I try to preview the script, appear the message: cannot render the file. If i use the old QTGMC the preview start. I have updated all the dll of the core plugins to the version requested by the avsi file. This is the part of the script relative to QTCMG

#Deinterlace
Loadplugin("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\SysInfo0129\SysInfo64.dll")
Loadplugin("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\vinverse\vinverse.dll")
Loadplugin("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\vsTCanny\vsTCanny.dll")
Loadplugin("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\RgTools\RgTools.dll")
Loadplugin("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\nnedi3\nnedi3.dll")
Loadplugin("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\masktools\masktools2.dll")
Loadplugin("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\mvtools\mvtools2.dll")
Import("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\Scripts\QTGMC.avsi")
video=AssumeTFF(video).QTGMC(Preset="Medium",FPSDivisor=1)

[Dogway]

Hi...
Dogway, please can you help me?
I'm trying to use your latest version of QTGMC with Ripbot, but when I try to preview the script, appear the message: cannot render the file. If i use the old QTGMC the preview start.

What AviSynth+ version do you use? I recommend using latest v3.7.2 test3 (https://forum.doom9.org/showthread.php?t=181351). And redownload QTGMC (https://github.com/Dogway/Avisynth-Scripts/blob/master/MIX%20mods/QTGMC.avsi) just in case, as I had a typo yesterday.

[salvo00786]

Ok, I downloaded the new file, but same error. I use Ripbot not Avisynth. Can you help me on Ripbot?

[Dogway]

It uses AviSynth+ as the backend, but it is still using the version from one year ago.
Simply install latest AVS+ version or replace by hand "%PROGRAMFILES%\AviSynth+".

I also modified the RipBot installer so simply save it to the extracted folder of "Avisynth_3.7.2_20220208_test3.7z" and execute the file, to let it do automatically.

modified install.cmd:
@ECHO OFF

cd /d %~dp0

ECHO AviSynth+ MT (r3622 x86/x64) Installer by Atak_Snajpera (modified)
ECHO =======================================================

SET HasAdminRights=0
FOR /F %%i IN ('WHOAMI /PRIV /NH') DO (
IF "%%i"=="SeTakeOwnershipPrivilege" SET HasAdminRights=1
)

IF NOT %HasAdminRights%==1 (
ECHO Error:No admin rights detected.
ECHO This script must be run as an Administrator.
ECHO.
GOTO END
)

if "%ProgramW6432%" == "" (
SET InstallPath="%PROGRAMFILES%\AviSynth+"
) else (
SET InstallPath="%PROGRAMFILES(x86)%\AviSynth+"
)

if "%ProgramW6432%" == "" (

xcopy "x86\plugins" "%InstallPath%\plugins+\" /Y
xcopy "x86\AviSynth.dll" "%SYSTEMROOT%\System32\" /Y
xcopy "x86\system\DevIL.dll" "%SYSTEMROOT%\System32\" /Y

reg add HKEY_LOCAL_MACHINE\SOFTWARE\Avisynth /ve /d %InstallPath% /f
reg add HKEY_LOCAL_MACHINE\SOFTWARE\Avisynth /v plugindir2_5 /d %InstallPath%\plugins /f
reg add HKEY_LOCAL_MACHINE\SOFTWARE\Avisynth /v plugindir+ /d %InstallPath%\plugins+ /f
ECHO =================================================================

) else (
xcopy "x86\plugins" "%InstallPath%\plugins+\" /Y
xcopy "x64\plugins" "%InstallPath%\plugins64+\" /Y
xcopy "x64\AviSynth.dll" "%SYSTEMROOT%\System32\" /Y
xcopy "x86\AviSynth.dll" "%SYSTEMROOT%\SysWOW64\" /Y
xcopy "x64\system\DevIL.dll" "%SYSTEMROOT%\System32\" /Y
xcopy "x86\system\DevIL.dll" "%SYSTEMROOT%\SysWOW64\" /Y

reg add HKEY_LOCAL_MACHINE\SOFTWARE\Avisynth /ve /d %InstallPath% /f
reg add HKEY_LOCAL_MACHINE\SOFTWARE\Avisynth /v plugindir2_5 /d %InstallPath%\plugins64 /f
reg add HKEY_LOCAL_MACHINE\SOFTWARE\Avisynth /v plugindir+ /d %InstallPath%\plugins64+ /f

reg add HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Avisynth /ve /d %InstallPath% /f
reg add HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Avisynth /v plugindir2_5 /d %InstallPath%\plugins /f
reg add HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Avisynth /v plugindir+ /d %InstallPath%\plugins+ /f
ECHO =================================================================
)

:END
rem pause

[salvo00786]

ok, thanks but:
1)When I try to install avs+ the program is in directory program files x86 and not in program files
2)I can find only 7z portable version of ripbot. Where is your modded version?
3)I downloaded Avisynth_3.7.2_20220208_test3.7z and i created install cmd with your text, but when i extract Avisynth_3.7.2_20220208_test3.7z, in the root directory of 7z, your file doesn't start

[salvo00786]

I tried to copy manually the x86 plugins in RipBot264v1.26.0\Tools\AviSynth\AviSynth+\plugins+ and x64 plugins in RipBot264v1.26.0\Tools\AviSynth\AviSynth+\plugins64+. The same operation in the C:\Program Files (x86)\AviSynth+ directory. Now when I try to start the script this error appear

[guest]

ok, thanks but:
1)When I try to install avs+ the program is in directory program files x86 and not in program files
2)I can find only 7z portable version of ripbot. Where is your modded version?
3)I downloaded Avisynth_3.7.2_20220208_test3.7z and i created install cmd with your text, but when i extract Avisynth_3.7.2_20220208_test3.7z, in the root directory of 7z, your file doesn't start

Hi salvo00786,

At last, another RipBot user :)

How long have you been using it ??

Have you "created" many scripts before ??

Why I ask is, how do you "implement" your QTGMC script ???, do you copy it into the script window of RipBot ??, it's just that without trying your script myself, I can see at least 2 major problems.

Oh, and there is an easier way of "installing" Avisynth so that RipBot can use it...

[salvo00786]

I haven't created many scripts before. I have user Ripbot many times but only for deinterlacing or for denoising files with KNLMEANSCL. But I have used the integrated QTGMC until now. Now i want to use the latest version. Yes, I copy the scripts under the Deinterlace section of Script Windows in Ripbot. Please tell me what are these 2 major problems and the easier way of installing avisynth so that Ripbot can use it

[guest]

I haven't created many scripts before. I have user Ripbot many times but only for deinterlacing or for denoising files with KNLMEANSCL. But I have used the integrated QTGMC until now. Now i want to use the latest version. Yes, I copy the scripts under the Deinterlace section of Script Windows in Ripbot. Please tell me what are these 2 major problems and the easier way of installing avisynth so that Ripbot can use it

Thanks for your quick reply...

Not sure how often you "visit" the RipBot forum, but RipBot is definitely dying a slow death, unfortunately.

Like I said, I haven't tested your script, so I don't want to say anything right now, but like I said, I can see 2 errors, but will confirm.

I don't use QTGMC much, as I have very little interlaced material to work with.

So I will see what I can come up with and get back to you either here or a PM.

[salvo00786]

ok, thanks

[guest]

ok, thanks

OK, I think I've got it sussed...

You need to be VERY, VERY particular with the "calls" within the script, meaning their EXACT location !!!

So copy this into Notepad, save as an .avs, not .txt.

#Custom
Loadplugin("%AVISYNTHPLUGINS%\dfttest\dfttest.dll") may not need this
Loadplugin("%AVISYNTHPLUGINS%\SysInfo_0129\SysInfo64.dll")
Loadplugin("%AVISYNTHPLUGINS%\vinverse\vinverse.dll")
Loadplugin("%AVISYNTHPLUGINS%\vsTCanny-1.1.5\vsTCanny.dll")
Loadplugin("%AVISYNTHPLUGINS%\RgTools\RgTools.dll")
Loadplugin("%AVISYNTHPLUGINS%\nnedi3\nnedi3.dll")
Loadplugin("%AVISYNTHPLUGINS%\masktools\masktools2.dll")
Loadplugin("%AVISYNTHPLUGINS%\mvtools\mvtools2.dll")
Import("%AVISYNTHPLUGINS%\scripts\extras\ExTools76.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\SMDegrain347d.avs")
Import("%AVISYNTHPLUGINS%\scripts\extras\Zs_RF_Shared.avs")
Import("%AVISYNTHPLUGINS%\Scripts\QTGMC.avs")
video=AssumeTFF(video).QTGMC(Preset="Medium",FPSDivisor=1)

Save it into the Custom Folder under Tools / Avisynth plugins / Scripts / Custom..

You'll have to download ExTools, SMDegrain & Zs_RF_Shared and create a new folder called extras (as shown in script)

Then from in RipBot you can just choose it from the Custom Scripts "list", check with AVSMeter.

orig

#Deinterlace
Loadplugin("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\SysInfo0129\SysInfo64.dll")
Loadplugin("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\vinverse\vinverse.dll")
Loadplugin("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\vsTCanny\vsTCanny.dll")
Loadplugin("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\RgTools\RgTools.dll")
Loadplugin("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\nnedi3\nnedi3.dll")
Loadplugin("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\masktools\masktools2.dll")
Loadplugin("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\mvtools\mvtools2.dll")
Import("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\Scripts\QTGMC.avsi")
video=AssumeTFF(video).QTGMC(Preset="Medium",FPSDivisor=1)

------------------------------------------------------------

Ok, so for Avisynth, extract the "test 3" .7z with WinRar.

From there go to the x64 & x86 folders..

Copy the AviSynth.dll & DevIL.dll (from the system folder), into the x64 & x86 folders, respectively, under "Tools / Avisynth / Libraries" folder in RipBot main folder.

Then run the "install.cmd" in the main Avisynth folder, as Admin.

Good luck

[guest]

Error with ExTools v7.6, from v7.5, with DFTTest (for prefilter=4)

Error at Line 251 in SMDegrain v3.4 7d

[pinterf]

But I will wait until you add selective frameprop copy if you still plan to add it.
Yes, probably next week second half.

[salvo00786]

Hi...
TDS, thanks for your reply.
This is my complete script

Loadplugin("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\vinverse\vinverse.dll")
Loadplugin("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\SysInfo0129\SysInfo64.dll")
Loadplugin("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\SysInfo0129\SysInfo32.dll")
Loadplugin("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\vsTCanny\vsTCanny.dll")
Loadplugin("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\RgTools12\RgTools.dll")
Loadplugin("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\NNEDI309460\nnedi3.dll")
Loadplugin("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\masktools22230\masktools2.dll")
Loadplugin("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\mvtools2745\mvtools2.dll")
Import("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\Scripts\SMDEGRAIN347d.avsi")
Import("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\Scripts\Zs_RF_Shared1157.avsi")
Import("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\Scripts\ExTools.avsi")
Import("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\Scripts\QTGMC.avsi")
video=AssumeTFF(video).QTGMC(Preset="Slower")

I can tell you that all these strings are correct because i have all these plugin in the directories that you can see in my script.

1)I renamed vsTCanny-1.1.5 to vsTCanny, SysInfo_0129 to SysInfo0129.
2)All my scripts are in the scripts directory, and not in extra.
3)The new QTGMC in QTGMC.avsi, if I write QTGMC.avs, the program use the old QTGMC integrated.

[salvo00786]

Ok. Now it works. The resolutive step was the installation of the avisynth, thanks.

Another question. When I copied the dlls, for the 64bit dlls I used the dlls in the x64_CUDA_enable directory instead of the x64 directory, because I have a RTX3070 card. Is this correct?

[guest]

Hi...
TDS, thanks for your reply.
This is my complete script

Loadplugin("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\vinverse\vinverse.dll")
Loadplugin("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\SysInfo0129\SysInfo64.dll")
Loadplugin("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\SysInfo0129\SysInfo32.dll")
Loadplugin("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\vsTCanny\vsTCanny.dll")
Loadplugin("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\RgTools12\RgTools.dll")
Loadplugin("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\NNEDI309460\nnedi3.dll")
Loadplugin("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\masktools22230\masktools2.dll")
Loadplugin("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\mvtools2745\mvtools2.dll")
Import("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\Scripts\SMDEGRAIN347d.avsi")
Import("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\Scripts\Zs_RF_Shared1157.avsi")
Import("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\Scripts\ExTools.avsi")
Import("A:\PORTABLE\RipBot264v1.26.0\Tools\AviSynth plugins\Scripts\QTGMC.avsi")
video=AssumeTFF(video).QTGMC(Preset="Slower")

I can tell you that all these strings are correct because i have all these plugin in the directories that you can see in my script.

1)I renamed vsTCanny-1.1.5 to vsTCanny, SysInfo_0129 to SysInfo0129.
2)All my scripts are in the scripts directory, and not in extra.
3)The new QTGMC in QTGMC.avsi, if I write QTGMC.avs, the program use the old QTGMC integrated.

OK, if that's how you want to have it setup, that's entirely up to you.

I use the "extra" folder, as I have added so many other scripts to RipBot, some of them have the same name, and they can't be in the same folder.

I don't use avsi, they should be avs, for RipBot, but again, if they work for you...good :)

As for QTGMC, you'll have to put it where the "old" one is, just rename the original, so that the process uses Dogway's build.

[guest]

Ok. Now it works. The resolutive step was the installation of the avisynth, thanks.

Another question. When I copied the dlls, for the 64bit dlls I used the dlls in the x64_CUDA_enable directory instead of the x64 directory, because I have a RTX3070 card. Is this correct?

So it obviously needed the newest version :)

I don't use the CUDA version, but if it works for you, then why not...you can always "test" the non CUDA ones.

I did ask that very same question, I'll see if I can find it, and let you know.

https://forum.doom9.org/showthread.php?p=1955585#post1955585

[Dogway]

Error with ExTools v7.6, from v7.5, with DFTTest (for prefilter=4)

Error at Line 251 in SMDegrain v3.4 7d

Yes, noticed yesterday, a stupid typo in ExTools, updating later today with some other features.


EDIT: Updated ExTools, besides the fix I improved ex_corner() temporal coherence for that I needed a motion mask, since I didn't want to go to the MVTools route I made a new motion mask filter called ex_occlusion() works well as a MMask replacement and it's very fast.

[gispos]

@Dogway, please give your versions different names than the 'original'.

For example In your version of 'QTGMC' other parameters and functions are used. And it's also kind of misleading (what am I using right now).
And also the sub functions should be renamed from QTGMC.

I also can't create a function definition in AvsPmod for two times QTGMC, so I can only ban your version completely.
As default I use the normal QTGMC, it is faster for me.

I am so cheeky:). Dogway's QTGMC renamed to ex_QTGMC, also sub functions. (2. upload)
A few places sub function rename forgotten and AvsPmod function definetion added.
https://www.mediafire.com/file/id2q3qi044qa95u/ex_QTGMC.7z/file

[kedautinh12]

QTGMC changed name to STGMC by real.finder. I think you need change your QTGMC to STGMC. Why you need the developer of QTGMC from 2015 like Dogway need change his QTGMC to other name?? I think the person need change QTGMC to other name is you
https://github.com/realfinder/AVS-Stuff/blob/Community/avs%202.6%20and%20up/STGMC.avsi

[Dogway]

@gispos what version is your 'original' QTGMC? My version is called QTGMC mod, you can name it like so since old version is outdated already (circa 2011) so I don't think there should be a name conflict. In 2012 I implemented 16-bit processing with Dither Tools and from 2015 real.finder took over, maybe you want to use STGMC (https://github.com/realfinder/AVS-Stuff/blob/Community/avs%202.6%20and%20up/STGMC.avsi)?

The main reason Expr() is slower than MaskTools is explained in this (https://forum.doom9.org/showthread.php?p=1943743#post1943743)post. The reason it's almost on par with real.finder QTGMC is because I optimized the filter to the extreme (merging expressions, wiser maths), same for LSFmod and other mods.

[guest]

Yes, noticed yesterday, a stupid typo in ExTools, updating later today with some other features.


EDIT: Updated ExTools, besides the fix I improved ex_corner() temporal coherence for that I needed a motion mask, since I didn't want to go to the MVTools route I made a new motion mask filter called ex_occlusion() works well as a MMask replacement and it's very fast.

Nice, fixed, thanks.

Not sure what to do with all the other options within your scripts.

[Dogway]

Nice, fixed, thanks.

Not sure what to do with all the other options within your scripts.

What options?

[guest]

What options?

Maybe "options" was the wrong term, but if I look thru all the many, many lines of code, etc, there just seems (to me) that there are other filters & codes that maybe used as a stand alone.

eg:- ex_corner() and ex_occlusion()

or are these just part of the "big picture" of the main script ??

Forgive me, this is the "noobie" in me coming out.

[gispos]

QTGMC changed name to STGMC by real.finder. I think you need change your QTGMC to STGMC. Why you need the developer of QTGMC from 2015 like Dogway need change his QTGMC to other name?? I think the person need change QTGMC to other name is you
https://github.com/realfinder/AVS-Stuff/blob/Community/avs%202.6%20and%20up/STGMC.avsi

@gispos what version is your 'original' QTGMC? My version is called QTGMC mod, you can name it like so since old version is outdated already (circa 2011) so I don't think there should be a name conflict. In 2012 I implemented 16-bit processing with Dither Tools and from 2015 real.finder took over, maybe you want to use STGMC (https://github.com/realfinder/AVS-Stuff/blob/Community/avs%202.6%20and%20up/STGMC.avsi)?

The main reason Expr() is slower than MaskTools is explained in this (https://forum.doom9.org/showthread.php?p=1943743#post1943743)post. The reason it's almost on par with real.finder QTGMC is because I optimized the filter to the extreme (merging expressions, wiser maths), same for LSFmod and other mods.
I'm probably behind the times. But it completely eludes me the logic.
QTGMC has been renamed to STGMC and the Dogway's full modeled version is now called QTGMC. What a mess.

I have hundreds of scripts with the original videos on CD and DVD, all old movies, many scripts contain QTGMC...


My version.

#-------------------------------------------------------------------#
# #
# QTGMC 3.33 by Vit, 2012 #
# QTGMC 3.382 2021 mod by A.SONY, based on v3.33d by Dogway #
# #
# A high quality deinterlacer using motion-compensated temporal #
# smoothing, with a range of features for quality and convenience #
# Originally based on TempGaussMC_beta2 by Didée #
# #
#-------------------------------------------------------------------#
#
# Full documentation is in the 'QTGMC' html file that may comes with this script, or even better see avisynth wiki
#
# --- LATEST CHANGES ---
#
# v3.382s
# - fix for n16 family
#
# v3.380s based on v3.34d (mod) by Dogway
# - Added Refinemotion Option for more accurate denoising.


This is from version v3.382s, Information is missing in STGMC 3.385, Has something changed?

# --- REQUIREMENTS ---
#
# Input colorspaces: YV12, YUY2, others as plugins supports
#
# Core plugins:
# MVTools2 (2.7.26 or above)
# MaskTools2 (v2.2.17 or above)
# NNEDI3 (0.9.4.6 or above)
# RgTools
#
# Core scripts:
# Zs_RF_Shared.avsi
#
# Additional plugins and scripts:


Short test with STGMC went well, or are other things needed after all? How many children did QTGMC get now, is there also a QTGMCmod? :rolleyes:

[Dogway]

Short test with STGMC went well, or are other things needed after all? How many children did QTGMC get now, is there also a QTGMCmod? :rolleyes:

Two only, three if I change to QTGMCmod.

# QTGMC 3.382 2021 mod by A.SONY, based on v3.33d by Dogway #

QTGMC mod is my branch for HBD (and now also for frameprops plus other), and indeed if you compare STGMC with QTGMC mod my version is way faster.
I don't claim my branch to be the official Vit's QTGMC replacement, so if you are very bothered by it and still want to use the 2011 version I can rename it to QTGMCM(). But as I say, for 8-bit QTGMC mod is not slightly slower than STGMC because I did something wrong in the code, if it's slower report upstream to improve Expr().

Also function names weren't a problem back then, you used the same name as a drop-in replacement, specially if the filter development ceased, that was in fact the intention so all your hundreds of old scripts wouldn't run into issues. As far as I'm concerned real.finder can still use QTGMC() or SMDegrain().

If you insist I should change the function name of most my filters, namely: GrainFactory3mod, SMDegrain, DestripeH and V, LSFmod, GMSD, MDSI, vsSSIM, BBSIM, SSIM_downsample, _IQA_downsample, RoundHalfToEven, Deblock_QED, MSUDeCor, SmoothDeblockn, SmoothD2c, ChromaReconstructor, ChromaReconstructor_faster, SPresso, SeeSaw, SeeSawMulti, FineSharp, NonlinUSM, SlopeBend, DetailSharpen, MedianSharp, SSSharpFaster, SSSharp, ReCon, blah, MedSharp, SharpenComplex2, pSharpen, CASm, Soothe and mmod (probably more)

Maybe "options" was the wrong term, but if I look thru all the many, many lines of code, etc, there just seems (to me) that there are other filters & codes that maybe used as a stand alone.

eg:- ex_corner() and ex_occlusion()

or are these just part of the "big picture" of the main script ??

What can you use in replacement for ex_corner()? I wouldn't make something if I didn't see a value. AFAIK there's no corner detection filter in either AVS or VS (unless you use external libraries like OpenCV). As for motion masks, mt_motion/ex_motion is too primitive and MVTools' MMask() is too slow (5~15fps), ex_occlusion() runs at 470fps and worked well for my purpose, still it's first version so it can be improved.

If you don't write script filters ExTools might not be of much interest to you, it's basically a backend library to feed higher level filters. Readily usable filters are in EX/MIX mods or my other Packs.

[guest]

What can you use in replacement for ex_corner()? I wouldn't make something if I didn't see a value. AFAIK there's no corner detection filter in either AVS or VS (unless you use external libraries like OpenCV). As for motion masks, mt_motion/ex_motion is too primitive and MVTools' MMask() is too slow (5~15fps), ex_occlusion() runs at 470fps and worked well for my purpose, still it's first version so it can be improved.

If you don't write script filters ExTools might not be of much interest to you, it's basically a backend library to feed higher level filters. Readily usable filters are in EX/MIX mods or my other Packs.

Hi Dogway, No, I don't write script filters, because I wouldn't know where to start, but I doubt that I need to do anything like that, for what I do.

I use ExTools because the other filters, etc, that you create, this is a major dependency, so I just go with what works for me.

Why I asked the question was simply because I see all these other "functions" within SMDegrain, ExTools (for example), and I just wondered if they could be used individually, if I knew how to....that's all.

Sorry for the frustration, just keep doing what you're doing, it's awesome :)

Cheers

[gispos]

If you insist I should change the function name of most my filters, namely:
I don't want to answare directly to that.
Suppose I want to use the 'original' filter and not your version, the reason should not matter now.

I load my filters via an avsi, so for example NonlinUSM is also loaded. Now I make my settings, ok, try a filter from Dogway.
What happens now? My NonlinUSM will be replaced by your version as soon as I load your SharpenerPack.
And your NonlinUSM version gives other results.

Possibly other values or parameters are necessary to achieve the same result (must not, but can be).
This is a no go for me.

Using the same names leads to confusion and is misleading.

[coolgit]

Using the same names leads to confusion and is misleading.

I have to agree with Gispos here.

I have used some of your filters and it clashed with some filters already in my plugins folders and had to fanny around sorting it out. Eventually I had to remove them.

When newbies search for avisynth they will end up at this main official site
http://avisynth.nl/index.php/Main_Page
then
http://avisynth.nl/index.php/QTGMC
then
click on QTGMC.avsi and get 404 error. One can get to realfinder's page and see STGMC and either get confused or download it.

Correct me if i am wrong, the main difference between current QTGMC and yours is yours has high bit depth (HBD).
If so then why not use QTGMC_HBD like Stainlesss did with ScSelect_HBD. https://forum.doom9.org/showthread.php?t=182392

Realfinder can carry on using QTGMC. Avisynth site just need to add your filter in external filter page with documentation. That why it would be neat and tidy and won't scare the newbies away.

[real.finder]

my QTGMC (STGMC) also work with HBD

I don't mind if dogway keep the name if the function give same output as old one (which I think it do aside from SMDegrain which is not)

[coolgit]

OK that's good, thought it was 8 bit only.

The main avisynth site weblink needs to change QTGMC to STGMC and a new page for Dogway QTGMC. That's assuming the 'how to use documentation' is different.

Just checked your STGMC and can see it has been updated a bit.

[Dogway]

I don't want to answare directly to that.
Suppose I want to use the 'original' filter and not your version, the reason should not matter now.

I load my filters via an avsi, so for example NonlinUSM is also loaded. Now I make my settings, ok, try a filter from Dogway.
What happens now? My NonlinUSM will be replaced by your version as soon as I load your SharpenerPack.
And your NonlinUSM version gives other results.

Possibly other values or parameters are necessary to achieve the same result (must not, but can be).
This is a no go for me.

Using the same names leads to confusion and is misleading.
If it gives other results REPORT. Must of my ports are meant to sanitize and unify code not to redefine its purpose (unless inconsistency). Why do you need 2 or 3 versions of NonlinUSM? A few handful versions of mod4(), two versions of Destripe or SPresso, 7 versions of Deblock_QED(), and so on... See my point? that leads to confusion and is misleading.

You need to understand what I'm doing. Why do you need:
"x x y - x y - abs 0.0001 + / *"
to change sign when you can now do
"x dup y - sgn *"

Same with the hundreds and inefficient ways to blur (I have even seen blur(1).blur(1).blur(1).blur(1).blur(1)...), like in your HighlightLimiter function.
NonlinUSM uses an unpredictable and non-linear way of blurring (what (https://forum.doom9.org/showthread.php?p=1821106#post1821106) is 'rad' here anyway?):
bicubicresize(round(width()/rad/4)*4,round(height()/rad/4)*4).bicubicresize(width(),height(),1,0)
blurring more when small? Shouldn't you want to preserve the intention regardless of resolution? It should be the other way around (it was also reported (https://forum.doom9.org/showthread.php?p=1821019#post1821019) earlier). For that I need to go back to the original thread and find out what resolution it was tested on and fit a function to scale that to any resolution.
In this case I ran into several issues because I developed NonlinUSM at the same time I worked on ex_GaussianBlur, but this was my take (it will never match the above SoftCubic100 upscale but the goal is to minimize error):
ex_GaussianBlur(rad*0.83825-0.0631,pad=true)
To make it work at any resolution simply scale rad based on it
ex_GaussianBlur((w/720.)*rad*0.83825-0.0631,pad=true)

Another point is filters with different arg names for the same purpose. 'show', 'showsettings', 'debug'... I do unify those so you don't have to look up the function arg list to find out. Same with 'chroma', bool 'UV', int 'UV', etc. Others like 'th', 'thr', 'thres', 'threshold'... or 'lo' and 'hi' in my filters which are a staple instead of 'low' and 'high', or 'th1' and 'th2', 'b' and 'w', 'dark' and 'bright', etc.

So the question goes back to, why do you want to use old, unmaintained and slow filters and what can we do to improve it?

I'm open to change the name of those that intentionally changes the output though (despite for fixing or improving purposes).

my QTGMC (STGMC) also work with HBD

I don't mind if dogway keep the name if the function give same output as old one (which I think it do aside from SMDegrain which is not)

Didn't mean that, my point was (replying gispos) that despite QTGMC mod being a tiny bit slower than STGMC at 8-bit it's way faster for HBD which was it's main purpose for the mod to start with. I changed a few output related things some months back and plan to do further with Zopti findings so I'll be doing more changes.

I will change the name of QTGMC and LSFmod to start with since I added critical output improvement. Probably with 'plus' as suffix since now it supports also frameprops aside HBD.

[kedautinh12]

@Dogway I think you don't need change name when result don't different. And i say again why need Dogway change name of plugin when he develope it from 2015??
https://github.com/Dogway/Avisynth-Scripts/blob/361d4b3609e9da732248e3ec2cb059b308b9e383/MIX%20mods/QTGMC.avsi#L4

[coolgit]

Currently the avisynth site QTGMC link doesn't work since realfinder change his link to STGMC. My point is that newbies need simple access and working links. If dogway QTGMC is the current version then the links need to reflect that. Broken links put newbies off. Avisynth is hard to learn step by step and broken links isn't going to help.

Staxrip has the ability to update avs filters but can not access QTGMC at the moment. It seems there are some communication breakdown. Perhaps realfinder changed QTGMC to STGMC too early. Better coordination of changes to links would be the way to go.

[Dogway]

It's a matter of coming to an agreement. My stance was it was ok to keep function name but now if we agree (given gispos issues) we can change the name if output differs purposefully, doesn't matter how you reach there code-wise.
So in those terms for me real.finder can keep name and I will change those that changed behaviour like LSFmod or QTGMC. An exception for SMDegrain since I built it from the ground up basically.

If other filters' output are the same or very near the same, like Deblock_QED(), NonlinUSM() (should be identical to original, currently further refining it for resolution-aware), etc for me it's fine to keep function name the same, albeit calling the filter blabla mod or something. Maybe we can also do the same for abandon-ware, an example is ChromaReconstructor_faster, I modified it to have different bicubic coefficients but the purpose is the same. Same with SPresso(), output should be the same I just extended it to support HD sources but keeping the intention.

[real.finder]

Didn't mean that, my point was (replying gispos) that despite QTGMC mod being a tiny bit slower than STGMC at 8-bit it's way faster for HBD which was it's main purpose for the mod to start with. I changed a few output related things some months back and plan to do further with Zopti findings so I'll be doing more changes.

I will change the name of QTGMC and LSFmod to start with since I added critical output improvement. Probably with 'plus' as suffix since now it supports also frameprops aside HBD.

well, my QTGMC mod based on your old mod, also I did make it use dither_luma_rebuild by default so it also not 100% same output and speed as the original one, so I will keep it as STGMC

'plus' as suffix

this is the best option that will also let the user know that it is a version for avs+ only

[StvG]

I agree that the duplicated names should be changed especially if both versions are still maintained.
And moreover not only if the output is different but also if the requirements are totally different. For example when I use X filter and see a newer or modded filter with the same name I'm assuming that I can just download the new X filter and use it directly out of the box, but this is not the case here - we have to download a whole new set of requirements.

[Reel.Deel]

I agree that the duplicated names should be changed especially if both versions are still maintained.
And moreover not only if the output is different but also if the requirements are totally different. For example when I use X filter and see a newer or modded filter with the same name I'm assuming that I can just download the new X filter and use it directly out of the box, but this is not the case here - we have to download a whole new set of requirements.

I 100% agree with the requirement argument. If the "mod" script requires x amount of other supporting scripts/plugins to work then it should have a different name. I have always tried to maintain the QTGMC wiki page up-to-date but have not recently, not only because of other things I'm doing but because of the requirements changing. I remember when the script all of a sudden required SMDegrain to work, and then for YUY2 you needed PlanarTools (because converting to YV16 and and back to YUY2 at the end is waaaayyy too hard lol). For some other options you needed AnimeIVTC... and the list goes on. It will be one thing if at least it were documented, but sometimes it was not, and that is very off-putting and causes confusion. In those cases I think it's better for everyone if it has a different name.

TL;DR: if the modded script cannot be used as turnkey drop-in replacement, maybe it's best if it has a different name.

[StainlessS]

Yip, seems we are tryin' to emulate M$ DLL hell :)

DLL Hell:- https://en.wikipedia.org/wiki/DLL_Hell

[gispos]

@Dogway, I think it's great what you are doing and some things I really like. But please, I don't want to be forced to use all of your filters if I only want to use one.

With time (if you modify even more filters) it will become even more confusing. And filters are replaced in the background without my knowledge.
But I would like to determine myself what I use and certainly not be forced to do so. Or I have to rename the filters myself or copy them to another avs.

If you do not know, as was the case with me, you wonder what is suddenly going on and is an eternity looking for why the settings no longer fit.

[kedautinh12]

Dogway changed QTGMC to QTGMCp and LSFmod to LSFplus. When real.finder changed QTGMC to STGMC and don't change back QTGMC. I need download again old QTGMC which don't different STGMC, it's waste space for this. "Thank" anyone who request Dogway change name of QTGMC

[real.finder]

Dogway changed QTGMC to QTGMCp and LSFmod to LSFplus. When real.finder changed QTGMC to STGMC and don't change back QTGMC. I need download again old QTGMC which don't different STGMC, it's waste space for this. "Thank" anyone who request Dogway change name of QTGMC

avsi scripts take only few Kilobytes

[kedautinh12]

avsi scripts take only few Kilobytes

I'm obsessive-compulsive personality disorder (OCPD) :D

[Dogway]

I updated LSFmod to LSFplus and QTGMC to QTGMCp. For others I'm waiting for propCopy to support negative selection, since many of my next versions will fall in round numbers and I don't plan to touch them any further.
I took the last day to think about it, and I'm not sure you thought on the implications.

I 100% agree with the requirement argument. If the "mod" script requires x amount of other supporting scripts/plugins to work then it should have a different name. (...) I remember when QTGMC all of a sudden required SMDegrain to work, and then for YUY2 you needed PlanarTools (because converting to YV16 and and back to YUY2 at the end is waaaayyy too hard lol). For some other options you needed AnimeIVTC... and the list goes on. It will be one thing if at least it were documented, but sometimes it was not, and that is very off-putting and causes confusion. In those cases I think it's better for everyone if it has a different name.

TL;DR: if the modded script cannot be used as turnkey drop-in replacement, maybe it's best if it has a different name.

I make a great effort on documenting dependencies (not always sub-dependencies though).

...I'm assuming that I can just download the new X filter and use it directly out of the box, but this is not the case here - we have to download a whole new set of requirements.

Yip, seems we are tryin' to emulate M$ DLL hell :)

DLL Hell:- https://en.wikipedia.org/wiki/DLL_Hell

I'm not sure if you are for or against. Would you rather have heavy plugin+script fragmentation? ie. only for variable gaussian/binomial blur: GBlur2, ablur, BinomialBlur, GBlur, FastBlur, GaussianBlur, FRC_GaussianBlur42, FastGaussBlur. All unmaintained and/or slower than ex_gaussianblur() or vsTCanny(). Then each script doing the same task (like Highlightlimiter and ex_highlightlimiter() ) but you would rather keep VariableBlur.dll (for gaussianblur) and all the hundreds of duplicate plugins because different scripts require different plugins, having half a thousand files in the plugins directory with filters that do the exact same task. For me this is extremely confusing, specially for beginners.

In contrast for most of my mods the only dependencies are ExTools and ResizersPack, not adding this dependencies on top but shifting them from masktools, rgtools and the myriad of blurring filters. That means, most of my scripts are VERY low on dependencies (more so in EX mods) since ExTools already includes a bunch of low level filters like mean/gaussian/median/bilateral blurs and spatial/temporal denoisers.

To put it in perspective, this is a change of paradigm because most of my mods (even those I rescued from the death) will have to change name, same for real.finder mods which also have external dependencies (mostly Zs_RF_Shared), and now we will end with 3 script versions of the same filter and the end user being further confused. QTGMC, STGMC and QTGMCp. Is this ok?

______________

On another note I refined NonlinUSM() the fit was mostly fine (was fine in a previous commit), but on my reference script I had at the end:
mt_lutxy(o,"x 0 > x y ? ")
Which I tried to trace back but didn't find any reference. So I removed it from the expression and now the filter gives 1:1 match to original.
Now comes the resolution issue and I need feedback because there are two approaches, mimic what it does at low resolution (I assumed it is designed around 480p), or adapt it for HD resolution. Comment and uncomment and let me know which one do you think fits best for HD.
function NonlinUSM(clip o, float "str", float "z", float "pow", float "rad", float "ldmp", int "UV") {

rgb = isRGB(o)
w = width(o)
h = height(o)
bi = BitsPerComponent(o)
fs = propNumElements (o,"_ColorRange") > 0 ? \
propGetInt (o,"_ColorRange") == 0 : rgb

str = Default(str, 0.7) # strength
z = Default(z, 6.0) # zero point
pw = Default(pow, 1.6) # power
rad = Default(rad, 9.0) # radius for "gauss"
ldmp= Default(ldmp, 0.001) # damping for verysmall differences
UV = Default(UV, rgb ? 3 : 1)

z = ex_bs(z, 8, bi, fulls=true, flt=true)
ldmp = ex_bs(ldmp, 8, bi, fulls=true, flt=true)
zd = 1. / z
pw = 1. / pw

# g = o.ex_GaussianBlur((w/720.)*rad*0.79-0.122,pad=true) # This is to match 480p
g = o.ex_GaussianBlur((720./w)*rad*0.79-0.122,pad=true) # This is to adapt to HD sources (inversely proportional)

p = Format(pw == 1 ? "{str} * " : "{zd} * {pw} ^ {z} {str} * * ")
ex_lutxy(o, g, Format("x dup y - A@ abs "+p+" A dup * dup {ldmp} + / * A sgn * +"), UV=UV, fulls=fs)
}

[coolgit]

To put it in perspective, this is a change of paradigm because most of my mods (even those I rescued from the death) will have to change name, same for real.finder mods which also have external dependencies (mostly Zs_RF_Shared), and now we will end with 3 script versions of the same filter and the end user being further confused. QTGMC, STGMC and QTGMCp. Is this ok?


I do not have a problem if you replace QTGMC with an improved version, if the original is no longer worked on, but my issue is this link http://avisynth.nl/index.php/QTGMC should be updated simultaneously if the page is going to be redirected from realfinder to yours. Otherwise newbies will be confused.

What about a QTGMC pack with all the other relevant filters according to the requirements, so newbies and others can download one file to unzip rather than 5, 10 15, 20 or so files separately.

[kedautinh12]

yeah, that's right, many versions with QTGMC when Dogway changed name. I think newbie hard to understand when looking for avs in first time

[real.finder]

I do not have a problem if you replace QTGMC with an improved version, if the original is no longer worked on, but my issue is this link http://avisynth.nl/index.php/QTGMC should be updated simultaneously if the page is going to be redirected from realfinder to yours. Otherwise newbies will be confused.

I did update the wiki to freeze it on the last working QTGMC of my mod, but as I said, for people like gispos they have to set TV_range=false to get same output as Vit one
that all I can do for now

edit: @gispos check https://pastebin.com/raw/SYMD7ij5 the last update ever

[coolgit]

So when the time is right the present avisynth QTGMC link will direct to STGMC (still use TV_range=false) and a new page for updated dogway's QTGMC. Also http://avisynth.nl/index.php/External_filters deinterlacing section present QTGMC should be renamed STGMC (keeping description, plugin, colour format and author, the same) and a new addition for dogway's QTGMC (with new description, plugin, colour format and author).

Would the above be the best sensible way to go forwards?

[real.finder]

if gispos will be happy with https://pastebin.com/raw/SYMD7ij5 I will bring back the QTGMC to https://github.com/realfinder/AVS-Stuff and make the STGMC as a wrapper for QTGMC with TV_range as true by default

[coolgit]

Dogway QTGMC requirements.

ExTools can't be found on main avisynth site. Also where is v8.0 or above, best i have found is 7.7.
SMDegrain avisynth link is dead, 404.
Zs_RF_Shared not in avisynth external filter page.
AddGrainC v1.8.4 avisynth page version number needs updating.

[Dogway]

Dogway QTGMC requirements.

ExTools can't be found on main avisynth site. Also where is v8.0 or above, best i have found is 7.7.
SMDegrain avisynth link is dead, 404.
Zs_RF_Shared not in avisynth external filter page.
AddGrainC v1.8.4 avisynth page version number needs updating.

That's not my issue but the wiki maintainers so report to them. In my repo you will find all you need.
As I said above I was waiting for propCopy to update (updated today) so I can update the rest of the filters.

As for updating the names, it looks like there's no agreement? real.finder back to QTGMC, and keeping function names, same as I.

[coolgit]

That's not my issue but the wiki maintainers so report to them.

I would have thought those who create filters/plugins inform the maintainers rather than the maintainers lurking around forums on the off chance someone done an update, etc.

In my repo you will find all you need.
As I said above I was waiting for propCopy to update (updated today) so I can update the rest of the filters.

Does this updating the rest of the filters includes filters that QTGMC is relying on. I have downloaded all the necessary files and about to test your QTGMC and do some comparison. Should i hold off a bit until you finish your update or what?

As for updating the names, it looks like there's no agreement? real.finder back to QTGMC, and keeping function names, same as I.

I think what realfinder is doing is temporary to fix broken link on avisynth site. I could be wrong.