Dogway's Filters Packs [Archive] - Page 52

DTL

18th May 2023, 19:12

"Would you mind linking a source for that?"

You can run any software profiler and look at the timestamps around load from memory operations (in disassembly view). From freeware AMD uProf . Loading from memory is not any compute operation - it simply CPU core stall because no data to continue compute. But it is thread taking CPU time so Task Manager will show core load time (it not show real computing load - it only show % of time thread using core time).

Thread waiting for data can not free CPU core resources because thread switching is very costly operation. The only possible is in 'hyperthreading' mode when 2 threads uses 1 core resources the stalling thread waiting for memory may be switched to other thread if it got ready to compute data. So 'hyperthreading' sometime really make some total performance benefit.

The more advanced intel VTune will also show you lots of performance counters around cache misses and so on. Also VTune can create some hints about how much the application is memory-performance bounded (after analysis of memory-performance counters).

The real indirect way to estimate compute load is look into CPU Power usage - any real computing part make switching and in CMOS it takes some power. Stalling in idle parts in CMOS not draw power and not produce heat. So CPU engine put parts to idle zero power state if nothing to compute. So the more power draw to CPU and more heat produced - the more real useful computing is performed. Or at least some about useful shuffling of data in lots of caches (L1,L2,(L3,L4)). The AVX2 and AVX512 compute units running with good load draw lots of power and produce lots of heat (so typically CPU clock trotled to save from overheating).

CPU temp may be used to estimate how compute software is optimized for compute at current chip - with fixed cooling the CPU temp increases as software make more compute switching at given time. It not mean it is useful compute but at least not stall waiting for something.

"I have not encountered this under Win11 so far"

I still not see Win11 - may be Microsoft make some redesign of performance measurement tools in that version. If all supported CPU vendors provide required hardware performance counters. In old times OS can only measure thread time at core (OS knows start and end time of thread at core and can measure 'physical time' using common RDTSC instruction).

LeXXuz

18th May 2023, 19:52

The real indirect way to estimate compute load is look into CPU Power usage - any real computing part make switching and in CMOS it takes some power. Stalling in idle parts in CMOS not draw power and not produce heat. So CPU engine put parts to idle zero power state if nothing to compute. So the more power draw to CPU and more heat produced - the more real useful computing is performed. Or at least some about useful shuffling of data in lots of caches (L1,L2,(L3,L4)).

Well, looking at the temps and reported values for TDC, EDC and PPT, I'd say the CPU pretty much has its hands full and isn't really idling very much. :D

And between Prefetch 2,4,8,16, and 18...32 there's a quite good linear scale from 2 to 16, with almost double the prefetch = double the performance. After that the gradient drops with increasing prefetch but performance still increases and tops out with ~32. So I see no reason not to use 32 threads on a 16C/32T CPU.

I'll post a speed test between 'untouched', UHDHalf=true and UHDHalf=false with the 7950x soon for another comparison.

Dogway

18th May 2023, 20:12

@LeXXuz: I forgot to say, are you using DGSource() or a CPU bound loader? it makes substantial difference.

I think Xeon Max is enterprise tier, I'm looking forward Diamond Rapids w3 or w5, whatever delivers 8c/16t at least. And the thing with AVX512 is that not all are created equal, Intel now says they are going to bring it to mainstream line but does it include AVX512-VNNI?

DTL

18th May 2023, 21:50

I hope the Xeon Max (already renamed in several months after 'Sapphire Rapids HBM') is only one of many new families names in the future. May be end-user chip will be named i7-15xxx or more numbers. Or i9-15xxx, i11-xxxx and so on.

" 8c/16t at least."

Hyperthreading typically good only if it is close to free addition. Only after all other hardware resources are fully filled (max possible channels of RAM and max possible ways in the caches and so on). If the same priced less cores without HT but offer more DDR RAM channels - it may be faster with well optimized software. Also not very great optimized software may somehow benefit from more ways in the caches (Xeons may have 12..24 ways and low quality chips 8..12 ways at L2/L3). If the processing datasets fits in the L2/L3 caches.

"AVX512-VNNI?"

The only known real benefit from NN is may be deinterlacing with fighting special separated fields aliasing. And the interlaced content moves to the past now. I not know any other processing with real benefit from NN on CPU. May be if civilization will not die too quickly we can see 4:2:0 to 4:4:4 decoding of the same high quality as deinterlacing - but it may be visible to Die-Hard perfectionists only. The general benefit of AVX512 is 4x larger register file of AVX2 and 2x wider dispatch ports and lots of new faster instructions for simple integer 8/16 bit operations typically exist in F/BW/DQ/VL/VBMI old series. 202x years expected to be season of AVX512 most members in all CPU chips at the market but something go not as nice as expected. Also the software still very poorly optimized even for AVX2 and number of programmers with understanding in SIMD looks like fast shrinking too. So if we even have AVX512 everywhere - there already close to nobody knows how to program it. The 512 bytes AVX2 register file already not very easy to keep in mind about data placement - and AVX512 is 4x times larger. Only really nice (typically young) great brains can make nice fast handcrafted software for 2048 bytes data array and with complex computing (not simple 2x expanding of old poorly designed AVX2 software). Or with degrading and dying of real humans we need to some NN-robots to design nice AVX512 programs - still not exist may be.

Dogway

19th May 2023, 00:23

I think we need general NN support at CPU level as well since as we know the future is AI based and frees us up from those time-consuming/low-reward tasks.
My only real concern with future hardware is if they are going back to the efficiency road path as lately everything consumes and heats way over I consider stable.

By the way I updated TransformsPack with a new gamut compression function ported from Jed Smith:

ConvertFormat(cs_in="ACEScg",cs_out="709",EOTFi="",GC=true)

TM_Hable(mode="Dark",filmic=false)
CCTF("1886",false, tv_out=false)

ConvertBits(8,dither=1)

ACEScg - Linear
http://i.imgur.com/TmY8C8em.png (https://i.imgur.com/TmY8C8e.png)

709 - 1886
http://i.imgur.com/ikZuKucm.png (https://i.imgur.com/ikZuKuc.png)

709 - 1886 (+Gamut Compression)
http://i.imgur.com/TPJHjnkm.png (https://i.imgur.com/TPJHjnk.png)

Since probably all these samples are scene referred (radiance based) they look bland, lacking the filmic punch.
You can try setting 'filmic' to true for either Dark or Bright 'mode'. But I prefer the following.
Here's an example using LMT_DCP() (DCP Tone Curve OOTF)
TM_Hable(mode="Dark",filmic=false)
LMT_DCP()
CCTF("1886",false,tv_out=false)
709 - 1886+DCP OOTF (+Gamut Compression)
http://i.imgur.com/vtMSoXFm.png (https://i.imgur.com/vtMSoXF.png)

And here using LMT_EMoR() instead, a typical Camera Response Function fit. It's an OOTF+EOTFi so we suppress the 1886 inverse EOTF:
TM_Hable(mode="Dark",filmic=false)
LMTi_EMoR(tv_range=false)
709 - EMoR OOTF/EOTFi (+Gamut Compression)
http://i.imgur.com/kXJFFtHm.png (https://i.imgur.com/kXJFFtH.png)

EDIT: Same as above with LMT_EMoR() but with updated GamutCompression with non-linear integration of negative values
709 - EMoR OOTF/EOTFi (+Gamut Compression Updated)
http://i.imgur.com/5G2PTVnm.png (https://i.imgur.com/5G2PTVn.png)

LeXXuz

19th May 2023, 01:04

@LeXXuz: I forgot to say, are you using DGSource() or a CPU bound loader? it makes substantial difference.

No. I've used LSMASH. On purpose. The 7950x machine has no Nvidia GPU that can decode HEVC as I have no use for that.
Otherwise it wouldn't be a fair comparison between both systems.

The GT 710 is fine and fast enough for frameserving MPEG-2 and AVC. And the Intel ARC is great for OpenCL filters but doesn't support CUDA of course.
I usually don't do UHD content. For various reasons. This one was just for testing out of curiosity.

Anyhow, here is the same content encoded with a Ryzen 7950x system:

https://abload.de/thumb/speedtest_7950x_28zfph.png (https://abload.de/image.php?img=speedtest_7950x_28zfph.png)

Again, top-left to bottom-right:

tl: x265 medium preset CRF20, UHDHalf=true, Prefetch(16)
tm: x265 medium preset CRF20, UHDHalf=false, Prefetch(16)
tr: x265 medium preset CRF20, no filtering in Avisynth

bl: x265 medium preset CRF20, UHDHalf=true, Prefetch(32)
bm: x265 medium preset CRF20, UHDHalf=false, Prefetch(32)
br: x265 medium preset CRF20 AVX512, no filtering in Avisynth

UHDhalf=false benefits a little from Prefetch 32 over 16, while
UHDhalf=true is about the same. Already noticed this with more demanding scripts for 1080p too.

Lastly a comparison between default AVX2 and allowing AVX512. Difference is negligible, as I expected. But mileage may vary with different content.

Anyhow, I'm fine with these numbers.

Dogway

19th May 2023, 10:11

Thanks for the tests, the 7950x is a monster CPU so I think the numbers are fine, UHDHalf=true does almost a 63% perf increase so I think it's fine given people would resort to more convoluted scripts and filtering.
I don't know what filters do make use of AVX512 in AviSynth so I can't tell. My interest in the instruction is more for 3D DCC and games (if I ever happen to find time for that).

anton_foy

19th May 2023, 16:33

Dogway

22nd May 2023, 09:25

My footage in uhd is very sharp/finely detailed I guess and when using temporalsoften mode in uhdhalf=true it is considerably blurrier compared to uhdhalf=false. Not nearly as little difference as to LeXXuz comparison examples. Is there a way to regain the detail/sharpness by pel or something and yet gain the speed from mscalevector somewhat?
Can you post a before-after? TemporalSoften is very aggressive so it will make the shortcomings of vector scaling more apparent. You can try as mentioned earlier with limit=2 or something along those lines, and obviously 'LFR'. I think something is going on on the vector scaling and could be revisited on the mvtools2 side.

By the way I ported (https://github.com/Dogway/emulation-random/blob/master/MPC-HC/Shaders/DolbyVision.hlsl) a few functions like ICtCp_to_RGB(), GamutCompression() or TM_Hable() to an .hlsl shader to decode IPTPQc2 content on real time. It doesn't read the RPU so no dynamic tonemapping, MMR or poly reshaping but the output looks very fine.

anton_foy

24th May 2023, 16:52

Can you post a before-after? TemporalSoften is very aggressive so it will make the shortcomings of vector scaling more apparent. You can try as mentioned earlier with limit=2 or something along those lines, and obviously 'LFR'. I think something is going on on the vector scaling and could be revisited on the mvtools2 side.

By the way I ported (https://github.com/Dogway/emulation-random/blob/master/MPC-HC/Shaders/DolbyVision.hlsl) a few functions like ICtCp_to_RGB(), GamutCompression() or TM_Hable() to an .hlsl shader to decode IPTPQc2 content on real time. It doesn't read the RPU so no dynamic tonemapping, MMR or poly reshaping but the output looks very fine.

Yes probably so I want to try with the same footage with mdegrain mode also to see the difference but as for now and a while forward I haven't got access to my computer so I will post when I do. The limit param I need to experiment with too. Thanks!

Tempter57

27th May 2023, 09:56

Dogway

Dear master, will pay the attention that QTGMC 4.2 + mod causes appearance of artefacts round objects. It is forced to be rolled away back on QTGMC 4.00 + mod or I apply QTGMC v3.384s by A.SONY. I process in the core results of digitization old VHS interlaced camcoders and source DVC

Dogway

27th May 2023, 11:03

Dear master, will pay the attention that QTGMC 4.2 + mod causes appearance of artefacts round objects. It is forced to be rolled away back on QTGMC 4.00 + mod or I apply QTGMC v3.384s by A.SONY. I process in the core results of digitization old VHS interlaced camcoders and source DVC

Can you provide a small sample and the QTGMC+ call? I'm curious to know since v4.1 and v4.2 only implemented sharper base bob deinterlacers.

Tempter57

27th May 2023, 18:42

Can you provide a small sample and the QTGMC+ call? I'm curious to know since v4.1 and v4.2 only implemented sharper base bob deinterlacers.

Source https://www.dropbox.com/s/ctokgarx58c0sd0/la_rambla.avi?dl=0
Script
prefix="C:\Program Files (x86)\AviSynth+\"
AddAutoloadDir(prefix+"plugins64")

setmemorymax(8000)

DirectShowSource2("C:\Users\Alex\Downloads\la_rambla.avi", fps=25.000, preroll=15)

LoadVirtualDubPlugin("C:\Program Files (x86)\AviSynth+\plugins64\VirtualDub\ccd_64.vdf", "ccd", 0)

ChangeFPS(last,last,true) # initiate a small forward buffer

# pointresizing for lossless RGb <> YCrCb convertion
pointresize(last.width, last.height*2).ConvertToRGB32(interlaced=true, matrix="Rec709").pointresize(last.width, last.height)
separatefields() ## Remove some chroma noise with CCD in RGB (gives best results)
ccd(12,1)
weave()
pointresize(last.width, last.height*2).converttoYUV422(matrix="Rec709",interlaced=true).pointresize(last.width, last.height)

/*
converttoYUV422(matrix="Rec709",interlaced=true)
src=last
CCD(30)
CombinePlanes(src, last, src, planes="YUV")
*/

#==== Deinterlaced ====
AssumeBFF()
QTGMCp(Preset="placebo", NoiseProcess=0, threads=1)
#QTGMC(Preset="placebo", NoiseProcess=0, edithreads=1)
#QTGMCp(Preset="fast", sharpness=0.3, threads=1)
mergechroma(last)

stabplus(dxmax=16, dymax=16)
Crop(12, 4, -8, -0)

yahr().yahr()
# FineDehalo(rx=2.0, ry=2.0, thmi=80, thma=128, thlimi=50, thlima=100, darkstr=0, brightstr=1.0, showmask=0, contra=0.0, excl=true)
LSFmod(defaults="Fast", strength=40, Smode=3, Smethod=2, kernel=11, secure=false, Lmode=1, soft=0, soothe=true, keep=48, edgemode=1, edgemaskHQ=true)

deep_resize(960, 720, grain=0,qual=2)

SmoothCurve(Ycurve="0-0;16-16; 255-235", Ucurve="0-0;77-72;127-127;172-177;255-255", Vcurve="0-0;77-72;127-127;172-177;255-255", interp=100, limiter=false, TVrange=0, dither=-1)
SmoothLevels(0,0.96,255,0,255, useopt=0, HQ=true, useMT=1) # 0 -automatic choice, 1- SSE2 CPU, 2 - Sandy Bridge CPU

ex_unsharp(0.15).ex_unsharp(0.15,Fc=width()/1.5)

# ==== DEBANDING + ADDGRAIN ====
neo_f3kdb(15,64,42,42,0,0,sample_mode=2,dynamic_grain=true,keep_tv_range=true,dither_algo=3,mt=false)

gr = 0.3
size = 0.8
GrainFactory3mod(g1str=6*gr,g2str=8*gr,g3str=5*gr,g1size=1.4*size,g2size=1.3*size,g3size=1.2*size,g1cstr=1.5*gr,g2cstr=0.8*gr,g3cstr=0.5*gr,temp_avg=1,tv_range=true)

ConvertToYV12(interlaced=false, matrix="Rec709")

AssumeFPS(50.00)
Prefetch(12,24)
Compare result with variants QTGMC(Preset="placebo", NoiseProcess=0, edithreads=1) QTGMC v3.384s mod by A.SONY or QTGMC 4.00 + mod

Dogway

28th May 2023, 11:49

Thanks! I could see the issue, just tested the QTGMC+ part of your script. Basically it's a detail vs anti-moiree balance.
For the first half of the video everything looked fine but as soon as it couldn't resolve combing (artifact heavy motion areas) it created those issues.

I reverted the new default EdiModes to v4.0 except when the new arg 'moiree' is enabled.
In either case output now is more detail preserving than in v4.0.

Probably a good improvement would be to mask moiree areas so we could have the best of both worlds.

Check new version here (https://github.com/Dogway/Avisynth-Scripts/blob/master/MIX%20mods/QTGMC+.avsi).

Tempter57

28th May 2023, 15:22

Dogway
Unfortunately the problem with QTGMC 4.3 + mod remained. I have tried even to place QTGMC +. avsi in other folder, thought that the conflict of plug-ins is possible.

Dogway

28th May 2023, 16:42

Unfortunately the problem with QTGMC 4.3 + mod remained. I have tried even to place QTGMC +. avsi in other folder, thought that the conflict of plug-ins is possible.

Then I don't know what the issue you refer to. Mind posting a screenshot or something? The plugin versions required are listed on the top of the script.

Tempter57

28th May 2023, 22:34

Then I don't know what the issue you refer to. Mind posting a screenshot or something? The plugin versions required are listed on the top of the script.
Here three results of different versions QTGMC https://dropmefiles.com/e2wWu. The password of archive 123

Dogway

29th May 2023, 01:32

I tested encoding that portion and got no issues. Ensure the plugin versions and that you are on AVS+ 3.7.3 test9, I can't vouch for later git versions.

The problem however might be in the loader or what you process before the QTGMC+ call (I don't have the CCD plugin so I don't know).
Also it might be that DSS2 messes with frameprops. Try using FFMS2.
Also unrelated but there's no such thing as lossless RGB<->YCbCr, and less with point resize, you are matrixing so point resize will in some form leak into luma.
For lossless you can try YCoCgR but you need to upscale to 444 with an interpolation kernel.

Tempter57

29th May 2023, 04:47

DTL

29th May 2023, 08:38

"# pointresizing for lossless RGb <> YCrCb convertion
pointresize(last.width, last.height*2).ConvertToRGB32(interlaced=true, matrix="Rec709")"

With Rec709 matrix it can not be lossless because of clipping superwhites and superblacks. And with PC.709 matrix it was found required additional Levels() preprocessing for input YUV to make more correct RGB decoding - https://forum.doom9.org/showthread.php?p=1987708#post1987708 . It looks AVS still have lots of internal shadows of the past as amateur-designed software. So it hard to believe it can do something lossless enough. Also most of processing in 8bit interfilters data transfers adds quantization noise at each step.

" application DSS2"

DS is completely random source depending on the total system DS environment configured. So if user like to use DSS in any stable way it need to cure all OS DS environment (and keep it from any changes like even Windows update). Also developers can not check if bug happen remotely without getting and somehow running locally complete remote user DS OS configuration snapshot. It is close to impossible for private persons developers.

Dogway

29th May 2023, 12:11

I used AVS+ 3.7.3 test9.
The conflict reason really in incompatibility QTGMC + 4.1 mod and above with DSS2, and also filter VirtualDub ccd_64.vdf. At connection FFMS2 and removal in a script of the lines connected with ccd_64.vdf a problem disappears.
But after all truth that QTGMC + 4.0 mod and more low worked with DSS2. I admit, if the conflict has been caused only ccd_64.vdf, but with the advent of problems at application DSS2 I can not reconcile and I do not wish. I will be forced to refuse application QTGMC + 4.3 mod or you try to eliminate in the subsequent versions this defect.

Thanks for testing!
If DSS2 or other loader tags the clip as progressive or a different field order, that's a bug on the loader and not on my side.
AVS+ workflow is frame property based, if you want old school workflow you should switch to AviSynth 2.6 and use old tools otherwise you will run into numerous unexpected issues not only with my filters but also Asd-g plugins and filters.

Tempter57

29th May 2023, 12:39

Thanks you for explanations

salvo00786

30th May 2023, 20:06

Hi Dogway. I want to report that in your latest version of SMDegrain, the v4.4.0d, the prefilter 4 doesn't work, the program give an error. The other prefilters are ok.

Guest

31st May 2023, 02:27

Hi Dogway. I want to report that in your latest version of SMDegrain, the v4.4.0d, the prefilter 4 doesn't work, the program give an error. The other prefilters are ok.Line 322 (https://github.com/Dogway/Avisynth-Scripts/blob/master/SMDegrain/SMDegrain.avsi#L322)

No problems here, with existing script :(

Can you provide error message ?? (I know I forget to do that, from time to time)

salvo00786

31st May 2023, 07:37

Yes, this is the error

https://drive.google.com/file/d/1YeCP9ziRoOxjtr50GNkY1y4wImCjaHyi/view?usp=share_link

If I use the old SMDegrain instead of the latest, with the same settings, the error doesn't appear

kedautinh12

31st May 2023, 07:42

salvo00786

31st May 2023, 08:53

I am using the latest version of Staxrip. I have to check what version of avs+ is in the program.

salvo00786

31st May 2023, 09:58

Guest

31st May 2023, 11:20

I don't know if I am wrong, but I downloaded the latest avisynth+ from your link and I overwrited the files in the staxrip directory with the files downloaded. But when I use SMDegrain, the same error appear.

You may have to modify the call a little bit..

pre=ex_Median(mode="IQMST")
SMDegrain(tr=6,thSAD=600,thSADC=300,contrasharp=true,prefilter=pre,refinemotion=true)

Might be an idea to also ask the question on the StaxRip thread...

Dogway

31st May 2023, 12:20

@salvo00786: Update ExTools and try again. IQMST was a last addition, even so it's already a few months ago.

Guest

31st May 2023, 13:19

@salvo00786: Update ExTools and try again. IQMST was a last addition, even so it's already a few months ago.

Does StaxRip use or support Extools ???

Just checked, it's under the "x\Apps\Plugins\AVS\Scripts\ExTools.avsi"

At least v10.0, but latest here, is v10.2

madey83

31st May 2023, 14:17

Hello,

is there any way to improve details retantion of this call and also remove more background grain?

pre=ex_Median(mode="IQMST", UV=3,thres=15)
pre=ex_blend(pre,"blend",opacity=0.5).ex_median("IQMV").ex_sbr(1,UV=3)

my_TR = 3 # 4/5 depends on the grain
my_thSAD = 400
my_thSCD1 = my_thSAD + 100
my_thSADC = my_thSAD - 100

SMDegrain(mode="TemporalSoften", tr=my_TR, thSAD=my_thSAD, thSADC=my_thSADC, thSCD1=my_thSCD1, thSCD2=110, Str=2.0, limits=false, refinemotion=true, prefilter=pre, chroma=true, plane=4, search=5, subpixel=3)

madey83

31st May 2023, 14:26

Does StaxRip use or support Extools ???

Just checked, it's under the "x\Apps\Plugins\AVS\Scripts\ExTools.avsi"

At least v10.0, but latest here, is v10.2

i use Staxrip 2.18 and i aways import/load below:

import("E:\4k_Encode\Avisynth-Scripts-master\MIX mods\GradFun3plus.avsi")
import("E:\4k_Encode\Avisynth-Scripts-master\ExTools.avsi")
import("E:\4k_Encode\Avisynth-Scripts-master\SMDegrain\SMDegrain.avsi")
Import("E:\4k_Encode\Avisynth-Scripts-master\ResizersPack.avsi")
Import("E:\4k_Encode\Avisynth-Scripts-master\MIX mods\LSFplus.avsi")
Import("E:\4k_Encode\Avisynth-Scripts-master\MIX mods\SharpenersPack.avsi")
Import("E:\4k_Encode\Avisynth-Scripts-master\GradePack.avsi")
Import("E:\4k_Encode\Avisynth-Scripts-master\ScenesPack.avsi")
Import("E:\4k_Encode\Avisynth-Scripts-master\MasksPack.avsi")
Import("E:\4k_Encode\Avisynth-Scripts-master\MIX mods\Spresso.avsi")
import("E:\4k_Encode\Avisynth-Scripts-master\EX mods\DeblockPack.avsi")
import("E:\4k_Encode\Avisynth-Scripts-master\TransformsPack - Main.avsi")
import("E:\4k_Encode\Avisynth-Scripts-master\TransformsPack - Models.avsi")
import("E:\4k_Encode\Avisynth-Scripts-master\TransformsPack - Transfers.avsi")
loadplugin("E:\4k_encode\BM3DCUDA_AVS-test9\x64\BM3DCPU_AVS.dll")
loadplugin("E:\4k_encode\BM3DCUDA_AVS-test9\x64\BM3DCUDA_AVS.dll")
loadplugin("E:\4k_encode\BM3DCUDA_AVS-test9\x64\BM3D_VAggregate_AVS.dll")
loadplugin("E:\4k_encode\dgdecnv_244\DGDecodeNV.dll")
loadplugin("E:\4k_encode\vsTCanny-1.1.8\x64\Release\vsTCanny.dll")
loadplugin("E:\4k_Encode\neo_FFT3D_r11\x64\neo-fft3d.dll")
loadplugin("E:\4k_Encode\fmtconv-r30\win64\fmtconv.dll")
loadplugin("E:\4k_Encode\avs_libplacebo-1.1.4\x64\Release\avs_libplacebo.dll")
loadplugin("E:\4k_Encode\ClipClop_x86_x64_dll_v1-26_20181222\Avisynth+_x64\ClipClop_x64.dll")
loadplugin("E:\4k_Encode\DGHDRtoSDR_1.15\DGHDRtoSDR.dll")

and i do not have any problem with SMDegrain calls

Guest

31st May 2023, 15:10

i use Staxrip 2.18

I don't use StaxRip, but hopefully this help salvo...

StainlessS

31st May 2023, 15:30

Madey83,
loadplugin("E:\4k_encode\dgdecnv_244\DGDecodeNV.dll")
DgDecNv currently on v247.
https://www.rationalqm.us/dgdecnv/binaries/

Not sure, I think versions about 244 could produce garbled frame if jumping about within the clip,
but if step back a dozen or so frames and then stepping forward frame by frame, did not produce garbled frame on the original bad one.
Have not noticed similar behaviour recently, so is maybe fixed in v247.

EDIT: Actually, Seems it was updated yesterday, is now v248.

madey83

31st May 2023, 16:18

Madey83,

DgDecNv currently on v247.
https://www.rationalqm.us/dgdecnv/binaries/

Not sure, I think versions about 244 could produce garbled frame if jumping about within the clip,
but if step back a dozen or so frames and then stepping forward frame by frame, did not produce garbled frame on the original bad one.
Have not noticed similar behaviour recently, so is maybe fixed in v247.

EDIT: Actually, Seems it was updated yesterday, is now v248.

thx :)

Dogway

31st May 2023, 18:19

salvo00786

31st May 2023, 20:13

Hi... Thanks for all your replies. I tried with Staxrip 1.18 and I updated Extools, but the error is the same.

madey83

31st May 2023, 20:57

madey83: You are using very strong filters there, for instance IQMST paired with TemporalSoften. Without seeing your source this could be too much. Maybe use a milder prefilter and increase thSAD. Settings vary a lot from source to source.

Also there are updated versions of your plugins:
avs_libplacebo-1.2.0
BM3DCUDA_AVS-test10-cuda118
ClipClop_x86_x64_dll_v1-27_20210727

Hi Dogway,

my main problem is that i have not enough powerfull CPU on my PC, and i always try to get as much as possible clean picture (targeting bitrate between 2-3 Mbps for 1080p) with possible high details remain as possible and of course speed is important for me too. With above settings of SMDegrain + LSFplus + neo_f3kdb i got ~4 fps which is quite low in my opinion.

i have tried also below settings:

pre=STTWM(sw=50,tw=50,aw=100,sthres= 5,tthres=5) # Balanced

my_TR = 3 # 4/5 depends on the grain
my_thSAD = 500
my_thSCD1 = my_thSAD + 100
my_thSADC = my_thSAD - 100

SMDegrain(mode="STWM", limit=255, UHDHalf=true, prefilter=pre, tr=my_TR, thSAD=my_thSAD, thSADC=my_thSADC, thSCD1=my_thSCD1, thSCD2=110, LFR=false, limits=false, DCTFlicker=false, refinemotion=true, search=5, subpixel=3)

but remove grain was not enough. Rising TR and thSAD did not remove grain more... only reduced speed of encode.

this is the picture example where i would like to get clean background but have remain as much as possible details on foreground

https://postimg.cc/tZpLdbf8

DTL

31st May 2023, 21:48

Typically to visibly remove more grain and got less MPEG output bitrate at fixed crf-value it is require to make much larger tr-value adjustments (like doubling of tr-value only make photon shot noise about sqrt(2) lower). So playing around tr of 3,4,5 may be not any visible (also doubling of tr typically make processing fps half lower). To make some more visible step in subtle degraining it is more benefitical to move from linear FIR-type filter (like in current 2.7.45 MDegrainX) to IIR-type (also non-linear) filter (like pmode=2 in post 2.7.45 builds and VSTTempSmooth my tech demo). The IIR-type filter can remember best block or sample value over tr-scope and continue output it (equal to frame-duplicate but at block or sample level) and make most MPEG-friendly frame sequence to encode.

FIR-type filter linear because of its math can not reach such samples stability over many frames with not very large tr-values. And residual very small samples non-stability over sequence of frames to encode still create very high 'noise bitrate' to MPEG encoder.

So it may be recommended to wait until new processing modes will be 'officially' released in mvtools/vsttempsmooth (or sample-based vsttempsmooth-like processing will be also integrated as additional mode into MDegrainN as planned depending on some algorithm like for blocks with too bad SAD/dismetrc and so like). Not waste time and CPU/power to use too outdated mvtools manalyse/degrain of about 1..2 decade old. Even with some scripting enchancements over very old and simple FIR-type linear degrain engine. Playing around very simple manalyse/mdegrain engine of 200x years design can not make degraining better and better each year with some 'prefiters' shuffling and simply increasing tr and thSAD. The old 2.7.45-mvtools based denoiser may reach its limits already at 201x years.
I hope if pinterf will have time in 202x years we may still see updated mvtools version 'stable' of post-2.7.45 version number with many new features added. So Dogway will make redesign of SMDegrain for better degrain performance.

I understand users like to make encodings today and not wait years and decades for next better version of software - but things become slower and slower now. After very powerful design of opensource software in 200x years users may be used to have more and more advanced software releases several time per year and now in 202x as most of opensource at current planet and current EU+US civilization is died already it may be only 1 new updated release per decade or even more rare.

Dogway

31st May 2023, 23:13

madey83: Update a sample clip so I can have a look. Typically this is a matter of configuring a nice prefilter. What CPU do you have? So I can compare to mine which isn't top notch either.

madey83

1st June 2023, 05:58

madey83: Update a sample clip so I can have a look. Typically this is a matter of configuring a nice prefilter. What CPU do you have? So I can compare to mine which isn't top notch either.

@Dogway

Regarding my CPU it is ashamed to admit by it is i7 10750h....

example:
https://mega.nz/file/8QgQSTbQ#Y2XXFl4te93JnrZ1qBClec9e1VpEw3oSclOZDl1LoPs

this is my entire call which gives me bitrate ~3500 Kbps, and yes i know HDR/DV source should have DGHDRtoSDR but i can't properly find white point of my source and i think some colors are shifted/not match to the source. I will update this matter in next time, when i will see this...

edit:
ConvertBits(16)

deep_resize(1920,edge="Zopti1080")
MaxFALL=22 #check for DV/HDR white point

pre=DGHDRtoSDR(mode="pq",white=MaxFALL,gamma=1/2.4,tm=1.0).ex_Median(mode="IQMST", UV=3,thres=15)
pre=ex_blend(pre,"blend",opacity=0.5).ex_median("IQMV").ex_sbr(1,UV=3)

my_TR = 3 # 4/5 depends on the grain
my_thSAD = 400
my_thSCD1 = my_thSAD + 100
my_thSADC = my_thSAD - 100

SMDegrain(mode="MDegrain", tr=my_TR, thSAD=my_thSAD, thSADC=my_thSADC, thSCD1=my_thSCD1, thSCD2=110, Str=2.0, limits=false, refinemotion=true, prefilter=pre, chroma=true, plane=4, search=5, subpixel=3)
LSFplus(strength=50, preset="LSF", secure=true, Lmode=3, soft=-2, soothe=true, Smode=3, Smethod=3, Szrp=16, Spwr=4, SdmpLo=4, SdmpHi=48, overshoot=1, undershoot=1, Overshoot2=1, Undershoot2=1, keep=20, edgemode=0)
neo_f3kdb(preset="veryhigh/nograin", dynamic_grain=false, sample_mode=2,keep_tv_range=false,dither_algo=3,mt=false)
fmtc_bitdepth(bits=10, dmode=7)
Prefetch(10,12)

FYI: i'm newbie in avisynth filtering, so this is my level of understanding how to use it.... :)

LeXXuz

1st June 2023, 09:24

this is my entire call which gives me bitrate ~3500 Kbps, and

Well if you care to bound your encodes to a certain bitrate you may tweak your encoder settings instead of trying to denoise your sources too much.
You can't 'denoise' any source down to your desired bitrate without losing too much detail at some point.

Why not do 'some' denoising for better compression and opt for a 2-pass encode at ~3.5Mbit instead of CRF?

madey83

1st June 2023, 09:49

Hi LeXXuz,

If I would have enough powerful CPU then I would be possible to do 2 pass encode, but I don't have it and double the time for single episode it is to much for me. That's the reason why I prefer CRF with denoising + plus clean sharp picture is my preference... :)

LeXXuz

1st June 2023, 10:03

So it may be recommended to wait until new processing modes will be 'officially' released in mvtools/vsttempsmooth (or sample-based vsttempsmooth-like processing will be also integrated as additional mode into MDegrainN as planned depending on some algorithm like for blocks with too bad SAD/dismetrc and so like). Not waste time and CPU/power to use too outdated mvtools manalyse/degrain of about 1..2 decade old. Even with some scripting enchancements over very old and simple FIR-type linear degrain engine. Playing around very simple manalyse/mdegrain engine of 200x years design can not make degraining better and better each year with some 'prefiters' shuffling and simply increasing tr and thSAD. The old 2.7.45-mvtools based denoiser may reach its limits already at 201x years.
I hope if pinterf will have time in 202x years we may still see updated mvtools version 'stable' of post-2.7.45 version number with many new features added. So Dogway will make redesign of SMDegrain for better degrain performance.

I understand users like to make encodings today and not wait years and decades for next better version of software - but things become slower and slower now. After very powerful design of opensource software in 200x years users may be used to have more and more advanced software releases several time per year and now in 202x as most of opensource at current planet and current EU+US civilization is died already it may be only 1 new updated release per decade or even more rare.

You know, you have a point there I think.
SMDegrain is an excellent script, but after many month and hundreds of tested configurations, it always came down to the very same conclusion that mvtools is the weak spot here and in its current state couldn't cut it to satisfy my needs and expectations of a high-class motion-based denoising workflow on more demanding sources.

I'm almost at the brink of saying I'd better buy more hard drives instead of wasting that money on energy cost for half-satisfying 'remastering' jobs of my collection which I may regret one day.

And here is the big problem I see for any future development. 20 years ago disc space was expensive and hard drive space even more. So lots of people were looking for smarter solutions to back up their content. Hence began a vast development in filters and codecs to reduce file sizes while maintaining or even improving quality. I've been here long enough and witnessed most of that evolution.

Today, storage space is cheap. Talking TB now instead of GB two decades ago. In the same time storage cost went down, energy cost went up constantly. Making video transcoding less and less attractive to many people. In my circle of friends I'm the only one left still caring about encoding efficiency and video filtering. All the other guys do is backing up their discs to their storage networks and be done with it.

And here I am, still wasting lots of money on a somewhat outdated 'profession' with results that don't even fully convince me in the end. Makes you think.

I do hope mvtools will get the updates one day it really needs to greatly improve motion-based denoising methods to a level that can satisfy ambitious quality expectations.
But like you said, we need the people that have the time and knowledge to code and implement those things. And these people get more rare every year.

Don't want to get this OT and like I said, Dogways script is excellent but it can only do as much as its core element. It's like having a Porsche with an old VW engine. Looking awesome at first glance but gets more unsatisfying the deeper you look into it.

Guest

1st June 2023, 13:15

Hi... Thanks for all your replies. I tried with Staxrip 1.18 and I updated Extools, but the error is the same.

Have you tried StaxRip 2.19.0 ??

https://forum.doom9.org/showthread.php?p=1987631#post1987631

And did you try the script I suggested ??

https://forum.doom9.org/showthread.php?p=1987812#post1987812

DTL

1st June 2023, 21:02

And here I am, still wasting lots of money on a somewhat outdated 'profession' with results that don't even fully convince me in the end. Makes you think.

I do hope mvtools will get the updates one day it really needs to greatly improve motion-based denoising methods to a level that can satisfy ambitious quality expectations.
But like you said, we need the people that have the time and knowledge to code and implement those things. And these people get more rare every year.

If pinterf is currently have low time and busy with AVS core - the only known other good programmer is Asd-g with many AVS plugins supported and some supplementing with required SIMD/bitdepth and other features for today usage. May be open a petition to take mvtools after pinterf and add my partially implemented features to make bugfixed 'stable' build at least with most simple and usefull features added (of total current list about 40+ new features) ? Or may be even open crowdfanding project to collect some money to pay to good programmer to make new better build. After this the Dogway may make additional tests and finally update SMDegrain with new version of mvtools.

I tried to send request to Asd-g for maintaining mvtools versions after 2.7.45 of pinterf build with as most as possible of my additions and will wait for answer.