Avisynth+

[real.finder]

Parts of FFTW3 are not thread-safe

-Vit- did some changes back then to fix that https://forum.doom9.org/showthread.php?p=1423459#post1423459

[real.finder]

Yep, just debugging the same problem, it's inside dfttest.
I have fixed the problem.
Good news: the script is working fine.

A new version of dfttest will be needed, I'll ask DJATOM if he wishes to maintain it further or I can make the update.

Any more wishes on dfttest?

HBD and Y*, and if it possible to add something to make it easier/faster in motion compensation usage

[real.finder]

Prefetch (clip c, int threads, int "frames")

In the original Plus, you could use only one Prefetch, but you can use any number of CUDA versions.
Also, an argument has been added to specify the number of frames to prefetch.
Prefetch (1,4) # Make 1 thread stand and prefetch 4 frames
By doing so, flexible parallelization configuration is possible, such as pipeline parallelization.

threads
Number of threads. If it is 0, it passes without doing anything.

frames
Number of frames to prefetch.
Again, if it is 0, it passes without doing anything.


shouldn't Prefetch(threads=0) or Prefetch() did auto use of all cores/threads :) like old avs mt (http://avisynth.nl/index.php/MT#SetMTmode), and only if Prefetch(1) it passes without doing anything.

[StainlessS]

Possibly HDRAGC source code is needed which is unfortunately not available (never published)

Never say never:- http://www.mediafire.com/file/cgvipn95ssiiojt/hdragc_25_dll_20050521.zip/file

EDIT: Source is 0.1.5 alpha (which aint the latest. EDITED: )

Version changes:
Version 0.1.5 alpha - 21.05.2005:
- improvments - 4 times faster with significantly reduced halo artifacts (if you want old behaviour use parameter mode = 0)
Version 0.1.2 alpha - 30.04.2005:
- changed lightness distribution routine (less plastic look - midtones have higher contrast)
- new value for auto control of saturation
- calculated gain is independent of avg_lum parameter
Version 0.1.1 alpha - 29.04.2005:
- fixed bug with avg_lum (thanks to AVIL)
- small c optimization (thanks to tsp)
- new parameters (coef_gain, min_sat, circle)
Version 0.1 alpha - 25.04.2005 - first release


It has ASM, otherwise I probably would have had a go myself to convert to x64.

EDIT: Damn, not latest by a long way. Latest I have is Hdragc-1.8.7.zip.

EDIT: from v1.8.7

Version changes:
Version 1.8.6 - 09.09.2006
- added 'freezer' parameter - number of frame that is choosen to calculate gain parameters
Version 1.8.5 - 10.06.2006
- added auto shifting (black_clip parameter)
- avg_lum parameter is back
- new mode (mode = 2), which is using float precision for some calculations (best quality, now mode = 2 as default)
- fixed behaviour of coef_gain (for coef_gain > 1.0 it works as in previous versions)
- tweaked corrector (by default it's not a bit lower, so if you were using previously x, now try x + 0.1)
Version 1.8.1 - 13.05.2006
- fixed crash for black frame (thanks to Boulder)
Version 1.8 - 13.05.2006
- huge jump in version, huge jump in quality ;)
- improved old algorithm from version 0.2.1 (mode = 0), new algorithm (mode = 1) that even for high gains give natural picture
- added support for YV12 and YUY2, removed RGB32
- 3 times faster (but still without MMX, SSE, SSE2 optimizations)
- 10 times less memory usage
- better handling of saturation
- new parameters for sublte color correction (shift_u, shift_v)
Version 0.2.1 - alpha - 25.10.2005:
- fixed sharpening/denoising (they were turned off when gain was not applied to image)
- fixed protect mode
Version 0.2.0 - alpha - 23.10.2005:
- completely new lighting routin (sigma parameter not needed)
- new mode (mode = 2) in which radius is adaptively choosen, this removes artifacts as much as I was able to do ;). Slow, but faster and much, much better than mode = 0
- mode = 1 is again 2 times faster (and will be faster later, still no MMX, SSE optimization)
- new parameter - protect - for value = 1 it can help in some situation with blowing highlights (default on)
- new parameters - contrast, limit - for denoising/sharpening, almost for free - no slow down (experimental)
Version 0.1.5 alpha - 21.05.2005:
- improvments - 4 times faster with significantly reduced halo artifacts (if you want old behaviour use parameter mode = 0)
Version 0.1.2 alpha - 30.04.2005:
- changed lightness distribution routine (less plastic look - midtones have higher contrast)
- new value for auto control of saturation
- calculated gain is independent of avg_lum parameter
Version 0.1.1 alpha - 29.04.2005:
- fixed bug with avg_lum (thanks to AVIL)
- small c optimization (thanks to tsp)
- new parameters (coef_gain, min_sat, circle)
Version 0.1 alpha - 25.04.2005 - first release

[pinterf]

-Vit- did some changes back then to fix that https://forum.doom9.org/showthread.php?p=1423459#post1423459
I can't see it how.

[pinterf]

HBD and Y*, and if it possible to add something to make it easier/faster in motion compensation usage
So it supports 16 bit only through lsb?

[real.finder]

I can't see it how.

seems only plugins source in the Package, it has only the modded dll's of fftw3 and libfftw3f-3 here if you want to test https://www.solidfiles.com/v/p5eAgABYgRyVV

edit: I just tried, seems it not fix it, I get same as StainlessS


On x86 into VDub2 I get a sort of blank VDub2, nothing in frame, with 2.6GB ram use [just hangs, maybe some kind of recursive stack exhaustion].


but with mpc

So it supports 16 bit only through lsb?

yes, no one added HBD to it yet, there are vs port https://forum.doom9.org/showthread.php?t=171678 maybe it can help

[pinterf]

seems only plugins source in the Package, it has only the modded dll's of fftw3 and libfftw3f-3 here if you want to test https://www.solidfiles.com/v/p5eAgABYgRyVV

yes, no one added HBD to it yet, there are vs port https://forum.doom9.org/showthread.php?t=171678 maybe it can help

Anyway, dfttest itself had to be modded. Nothing to do with fftw3, but calling its plan should be thread safe even if the filter was set to MT_MULTI_INSTANCE.

[pinterf]

real.finder, StainlessS:

pls. test this one.
https://github.com/pinterf/dfttest/releases/tag/1.9.4.4

Just a quick update for MT test.

[StainlessS]

Well it dont crash at all for me, however, I've never actually ever used it, unless as a requirement of some script function,
dont even know how to use it. [Think I'm gonna have a nap, I'm a bit knackered].

[DJATOM]

I'll ask DJATOM if he wishes to maintain it further or I can make the update.
You're free to modify that plugin, I've lost the interest in avisynth once started to tinker with vapoursynth.
avs2yuv update is probably my last work on avisynth software, but who knows... for now avisynth is only useful for me when I have do deal with DVD content. Vapoursynth lacks some plugins, for example VFR mode in VDecimate.

[pinterf]

You're free to modify that plugin, I've lost the interest in avisynth once started to tinker with vapoursynth.
avs2yuv update is probably my last work on avisynth software, but who knows... for now avisynth is only useful for me when I have do deal with DVD content. Vapoursynth lacks some plugins, for example VFR mode in VDecimate.
Thanks, I saw that the repo is "archived"

[Reel.Deel]

@pinterf

Just in case you are not aware, here's an updated dfttest: https://github.com/299792458m/dfttest_mod

[real.finder]

real.finder, StainlessS:

pls. test this one.
https://github.com/pinterf/dfttest/releases/tag/1.9.4.4

Just a quick update for MT test.

quick test and seems not crash/freeze with my admfilter (https://github.com/realfinder/AVS-Stuff/raw/master/avs%202.5%20and%20up/Advanced%20Denoising.avsi)()

thank you for make avs better and better :)

even with https://forum.doom9.org/showpost.php?p=1904737&postcount=5245 it work!

[pinterf]

And a quick update
dfttest v1.9.5 with high bit depth support (and an lsb_in fix)

https://github.com/pinterf/dfttest/releases/tag/1.9.5

Now I'm going to have some rest.

edit: Reel.Deel, thanks, bookmarked. I'll look at it later. Just tell me if I forget about it.

[DJATOM]

Nice spotted, pinterf! I tested it on "upscaled" stacked frames so didn't even had a chance to trigger that bug :(

[real.finder]

And a quick update
dfttest v1.9.5 with high bit depth support (and an lsb_in fix)

https://github.com/pinterf/dfttest/releases/tag/1.9.5

Now I'm going to have some rest.

edit: Reel.Deel, thanks, bookmarked. I'll look at it later. Just tell me if I forget about it.

thanks, HBD seems work as quick test :thanks:

[pinterf]

And a final update to have AVX2 routines inside and a further fix
Download dfttest v1.9.6
https://github.com/pinterf/dfttest/releases/tag/1.9.6
(Since this dfttest issue emerged here, I won't open a separate topic for it now)

[Taurus]

@pinterf:
A big :thanks: to you!

[FranceBB]

Once again a big fat thank you for DFTTest as it's one of my favorite denoise filters!
Having native high bit depth is amazing so that I don't have to go back and forth between planar and stacked/interleaved anymore in my filterchain! :D
But most importantly, thank you for keeping 16bit stacked as well, as this way it's gonna be backward compatible with the old scripts as well. :)
I tested it and it works fine on my beloved Windows XP! :D (and it works flawlessly on my Win10 machine with a Xeon that I use at work to encode as well...)

Just a quick note for all of you who use the "opt" parameter:

BEFORE Ferenc's mod, we had:

opt -

Sets which cpu optimizations are used. Possible settings:

0 - auto detect
1 - c routines
2 - sse routines
3 - sse2 routines

default: 0


AFTER Ferenc's mod, we have:

opt -

Sets which cpu optimizations are used. Possible settings:

0 - auto detect
1 - c routines
2 - sse2 routines
3 - avx routines
4 - avx2 routines

There's no SSE now! So, if you're using Windows XP, please set it to 2 or 1 according to your CPU capabilities. Of course you won't be able to use AVX and AVX2 'cause they're not available on XP (even if your CPU supports them), therefore if you had "opt 3" in your old script (which meant SSE2), please make sure to change that to "opt 2" as it means SSE2 now; if you try to use the old "opt 3" it will try to use AVX assembly optimizations and it will not run unless you have Win7 and higher.

[StainlessS]

There's no SSE now! So, if you're using Windows XP, please set it to 2 or 1 according to your CPU capabilities.
Probably best to just use opt=0(auto) or better still just remove any opt arg.
Only force opt if absolutely necessary.

EDIT: Script stuff extracted from AvsInit.avsi, requires Grouchy2004 Sysinfo plug, parts use RT_stats, for RT_Ord().

Function IsWinXP() { Return Findstr(SI_OSVersionString,"Windows XP")!=0}

Function HasSSE() { S=SI_CPUExtensions i=S.Findstr("SSE") Return i!=0&&(S.RT_Ord(i+3)==0||S.RT_Ord(i+3)==RT_Ord(","))}
Function HasSSE2() { Return Findstr(SI_CPUExtensions,"SSE2")!=0}
Function HasSSE3() { Return Findstr(SI_CPUExtensions,"SSE3")!=0}
Function HasSSE41() { Return Findstr(SI_CPUExtensions,"SSE4.1")!=0}
Function HasSSE42() { Return Findstr(SI_CPUExtensions,"SSE4.2")!=0}
Function HasAVX() { S=SI_CPUExtensions i=S.Findstr("AVX")Return i!=0&&(S.RT_Ord(i+3)==0||S.RT_Ord(i+3)==RT_Ord(","))}
Function HasAVX2() { Return Findstr(SI_CPUExtensions,"AVX2")!=0}

[real.finder]

opt arg should be used only for testing, so end users should not use them

[pinterf]

Opt meaning was changed years ago. The bottleneck is fftw however, finding a properly fast build can help.

[real.finder]

there are still some random crash in realtime with MT

ColorBars(width=640, height=480, pixel_type="yv12")
admfilter(custom_filter="dfttest(Sigma=(adSigma+1.0)/f)")
Prefetch(4)

sometimes it crash from begin, but most of times it work until some random seeking then it crash

admfilter is in Advanced Denoising.avsi (https://github.com/realfinder/AVS-Stuff/blob/master/avs%202.5%20and%20up/Advanced%20Denoising.avsi) and it also need this (https://github.com/realfinder/AVS-Stuff/blob/master/avs%202.5%20and%20up/ZS%20AVS%20Shared%20Functions.avsi)

[pinterf]

there are still some random crash in realtime with MT

ColorBars(width=640, height=480, pixel_type="yv12")
admfilter(custom_filter="dfttest(Sigma=(adSigma+1.0)/f)")
Prefetch(4)

sometimes it crash from begin, but most of times it work until some random seeking then it crash


There is a default internal MT in dfttest. Could you try with threads=1 parameter, until I get to the topic?

[real.finder]

There is a default internal MT in dfttest. Could you try with threads=1 parameter, until I get to the topic?

well, I don't have avstp.dll in plugins folder (but I load it manually for heavy scripts), but yes threads=1 seems help, the random crash is become harder (with more than 35 random seek until crash) now, but it's still there

[LigH]

@StainlessS:

Will there be an AviSynth+ 3.5 CHM ... or is that Sisyphean labor?

[StainlessS]

It does sound a lot like rolling a big boulder up a hill - repeatedly. (I had to look that up)
lets see how things pan out.

[pinterf]

well, I don't have avstp.dll in plugins folder (but I load it manually for heavy scripts), but yes threads=1 seems help, the random crash is become harder (with more than 35 random seek until crash) now, but it's still there
Not avstp. Plugins with internal MT do not use necessarily avstp. Neither dfttest, nor JPSDR's plugins use it. And perhaps there are others as well.

[real.finder]

Not avstp. Plugins with internal MT do not use necessarily avstp. Neither dfttest, nor JPSDR's plugins use it. And perhaps there are others as well.

yes, I know that, but since dfttest was modded by Firesledge (aka cretindesalpes) who made avstp and add it to mvtools (was modded by him also) and dithertools

[wonkey_monkey]

Quick question: is the horizontally shifting chroma bug from <=2.61 fixed in Avisynth+?

http://avisynth.nl/index.php/Known_Issues#Resizers:_Chroma_Positioning_Bugs

[real.finder]

Quick question: is the horizontally shifting chroma bug from <=2.61 fixed in Avisynth+?

http://avisynth.nl/index.php/Known_Issues#Resizers:_Chroma_Positioning_Bugs

I think no for compatibility reasons, that why there are ResizeX script

[Selur]

sounds strange to keep a bug for compatibility reasons,...

[real.finder]

sounds strange to keep a bug for compatibility reasons,...

for avs+ there are this https://forum.doom9.org/showthread.php?t=173986

it should has no problem even with gamma

[Stereodude]

Am I misreading something or was high bit depth AVIsource support removed from Avisynth+ starting with build 2724?

from http://avisynth.nl/index.php/AviSource#10.2B_bit_inputs
Full support list (* only supported prior to 2724):

I have r2923 and can't open an AVI containing v210 format video with AVIsource. :confused:

[pinterf]

Am I misreading something or was high bit depth AVIsource support removed from Avisynth+ starting with build 2724?

from http://avisynth.nl/index.php/AviSource#10.2B_bit_inputs

I have r2923 and can't open an AVI containing v210 format video with AVIsource. :confused:
Could you check it with 3.5.0?
3.5.0 changelog:
Fix: crash when outputting VfW (e.g. VirtualDub) for YUV444P16, other fixes for r210 ...

[StainlessS]

Could you check it with 3.5.0?
Latest non test (Release) version v3.50:- https://github.com/AviSynth/AviSynthPlus/releases

[Stereodude]

Could you check it with 3.5.0?
3.5.0 changelog:
Fix: crash when outputting VfW (e.g. VirtualDub) for YUV444P16, other fixes for r210 ...

Doesn't work for P010 (saved by VD2) or v210 (saved by Resolve).

"AVISource: couldn't locate a decompressor for fourcc P010"

"AVISource: couldn't locate a decompressor for fourcc v210"

It does the same thing x64 or x86

[poisondeathray]

You would need a VFW codec for v210 to use AVISource() . Something like blackmagic codecs, drastic codecs, or AJA codecs. Matching x86, x64 versions for x86/x64 avisynth

Or use lsmash / ffms2

[Stereodude]

You would need a VFW codec for v210 to use AVISource() . Something like blackmagic codecs, drastic codecs, or AJA codecs. Matching x86, x64 versions for x86/x64 avisynth

Or use lsmash / ffms2
Wait, why can't it read an uncompressed AVI with data in those color formats?

lsmash/ffms2 has to index the entire file and I could do without a long delay to open the file (like 10 minutes for a 150GB file). My journey to find an export format from Resolve that is lossless and can by opened by Avisynth without indexing first continues.

[StainlessS]

Thanks PDR, I was gonna post (ask) about requirement for v210 decompressor aswell.
For anyone that wants a 10 Bit v210 AVI sample (10 secs), there was one posted here (Standard8_Scan_16FPS.avi by SnillorZ) :- https://forum.doom9.org/showthread.php?p=1875247#post1875247
NOTE 830MB, is BIG for a 10 secs clip.

EDIT: Presumably v210 Decompressor required despite uncompressed because above is 10 bit [not standard 8 bit YUV].

EDIT: I think I remember the time when was necessary to install eg HuffYUV or HelixYUV because Windows standard did not include a VFW decompressor/splitter for YV12.
EDIT: Or the Windows YUV decompressor was a bit broken.

[poisondeathray]

Wait, why can't it read an uncompressed AVI with data in those color formats?


I think it's a windows thing.

And some of those "professional" codecs actually send RGB when it's in a AVI container (maybe pixel_type="YUV422P10" can force navigation of proper 10bit 4:2:2, not sure)

AVI does not work very well for higher bit depths in very many programs.

MOV is supported much better for higher bit depths (even though it's an Apple-esque tied format it's preferable here because it works properly)


lsmash/ffms2 has to index the entire file and I could do without a long delay to open the file (like 10 minutes for a 150GB file). My journey to find an export format from Resolve that is lossless and can by opened by Avisynth continues.

You can use v210 in MOV container and that does not require indexing by Lsmash when you use LSmashVideoSource() .

[Stereodude]

Thanks PDR, I was gonna post (ask) about requirement for v210 decompressor aswell.
For anyone that wants a 10 Bit v210 AVI sample (10 secs), there was one posted here (Standard8_Scan_16FPS.avi by SnillorZ) :- https://forum.doom9.org/showthread.php?p=1875247#post1875247
NOTE 830MB, is BIG for a 10 secs clip.

EDIT: Presumably v210 Decompressor required despite uncompressed because above is 10 bit [not standard 8 bit YUV].
You can make a sample. It's really easy.

global OPT_Enable_V210 = true

colorbars(1280,720, "YUV422P10")
trim(0,499)Fast Recompress with VD2 to an AVI while dropping the audio it's 1.15GB (for 500 frames) which you can 7-zip down to all of 180kB.

[StainlessS]

Thanks very much for that SD, 180KB, very petite :)

EDIT: 800MB Standard8_Scan_16FPS.avi , compresses with 7zip v19.0 Ultra Compression = 483MB, not quite as good as colorbars compressed.

[Stereodude]

You can use v210 in MOV container and that does not require indexing by Lsmash when you use LSmashVideoSource() .
Okay, once I went digging in the lsmash thread and found the latest build it seems to work from a .mov without indexing. I had an old version before which returns garbled green video hence why I was using ffms2 for the .mov.

:thanks:

[pinterf]

I was testing Avisource inputs with Magicyuv codec and test files, Avisource could ask for planar rgbp10, r210 or R10k output for the same M0RG fourcc file.
Or P210, v210 or YUV422P10 for UQY2 or M0Y2 fourCC formats.

#AviSource("Sample_UQRA.avi",False) #rgb64
AviSource("Sample_UQRA.avi",False,"rgbp") #cannot req planar rgb output
#AviSource("Sample_UQRG.avi",False/*,"YV12"*/) #rgb48
#AviSource("Sample_UQRG.avi",False,"RGBP") #rgb planar, auto bit depth
#AviSource("Sample_UQY2.avi",False) YUV422P10

#AviSource("Sample_M0RG.avi",False)#.Histogram("levels",bits=10) # RGBP10
#AviSource("Sample_M0RG.avi",False,"RGBP")#.Histogram("levels",bits=10) # RGBP10
#AviSource("Sample_M0RG.avi",False,"RGB64")#.Histogram("levels",bits=10) # RGB64
#AviSource("Sample_UQY2.avi",False,"v210").Info()#.Histogram("levels",bits=10) # YUV422P10
#AviSource("Sample_M0Y2.avi",False,"v210").Info()#.Histogram("levels",bits=10) # YUV422P10
#AviSource("Sample_M0Y2.avi",False,"P210").Info()#.Histogram("levels",bits=10) # YUV422P10
#AviSource("Sample_M0Y2.avi",False,"P216").Info()#.Histogram("levels",bits=10) # YUV422P16
#AviSource("Sample_M0Y2.avi",False,"P010").Info()#.Histogram("levels",bits=10) # YUV420P10
#AviSource("Sample_M0Y2.avi",False,"P016").Info()#.Histogram("levels",bits=10) # YUV420P16
#AviSource("Sample_M0RA.avi",False/*,"RGBAP10"*/)#.Histogram("levels",bits=10) # RGB10
#AviSource("Sample_M0RG.avi",False,"r210")#.Histogram("levels",bits=10) # RGB
#AviSource("Sample_M0RG.avi",False,"R10k")#.Histogram("levels",bits=10) # RGB
#AviSource("Sample_M0RG.avi",False,"RGBP")#.Histogram("levels",bits=10) # RGB

#AviSource("Sample_M0G0.avi",False/*,"RGBP10"*/)
#AviSource("Sample_M0RG.avi",False,"r210")#.Histogram("levels",bits=10) # RGBP10
#AviSource("cbv308.avi",False,"v308") #YV24 aka YUV444P8
#AviSource("cbv308.avi",False,"v408") #YUVA444P8
#AviSource("cbv410.avi",False,"v410") #YUV444P10
#AviSource("cbv308.avi",False,"v410") #error

Info()
ConvertToRgb32()

[Stereodude]

I was testing Avisource inputs with Magicyuv codec and test files, Avisource could ask for planar rgbp10, r210 or R10k output for the same M0RG fourcc file.
Or P210, v210 or YUV422P10 for UQY2 or M0Y2 fourCC formats.
So it is supposed to be able to open uncompressed AVI's with those color formats or not?

[pinterf]

So it is supposed to be able to open uncompressed AVI's with those color formats or not?
For AviSource it depends on the installed codec. No codec, no video. (No GUS, no sound :) )

[StainlessS]

No codec, no video. (No GUS, no sound :) )
Bob Marley fan, eh. [EDIT: No woman, no cry]

[pinterf]

Demoscene, Future Crew, EMF and others :)