AviSynth+ thread Vol.2

[qyot27]

Presumably, the check exists the way it does because you would be allocating PLANAR_Y and PLANAR_G at the beginning of the sequence, since they're both the first plane in the the sequence order. Planar RGB is ordered GBR, as opposed to endian-specific packed RGB (which is either BGR in little or RGB in big).

For comparison, see how FFmpeg initiates the format in the AviSynth demuxer,
http://git.videolan.org/?p=ffmpeg.git;a=blob;f=libavformat/avisynth.c;h=e85b9ae48878e52df5bb31da450f58229dff5277;hb=95a6788314a2f5080e8d5488dd5ba1040abeba6f#l127

127 static const int avs_planes_rgb[3] = { AVS_PLANAR_G, AVS_PLANAR_B,
128 AVS_PLANAR_R };
129 static const int avs_planes_yuva[4] = { AVS_PLANAR_Y, AVS_PLANAR_U,
130 AVS_PLANAR_V, AVS_PLANAR_A };
131 static const int avs_planes_rgba[4] = { AVS_PLANAR_G, AVS_PLANAR_B,
132 AVS_PLANAR_R, AVS_PLANAR_A };

[ENunn]

Hey there. I don't know if this is the place for bug reports or not but I found a bug. Whenever I use the normalize function, the audio gets super loud. Even if I have it set to 0.0001 it's really loud. I downgraded to 3.7.3 final and it's fixed. Don't know if I'm the only one with this issue but I wanted to give you a heads up.

[tebasuna51]

Hey there. I don't know if this is the place for bug reports or not but I found a bug. Whenever I use the normalize function, the audio gets super loud. Even if I have it set to 0.0001 it's really loud. I downgraded to 3.7.3 final and it's fixed. Don't know if I'm the only one with this issue but I wanted to give you a heads up.

I can't reproduce the problem with last test 14 (r4066), what is your installed version?
If it is the same version please upload the .avs used and the source (or a sample)

[pwnsweet]

I'm also having this issue. I'd love to enjoy 3.7.3 but haven't been able to resolve this so I'm stuck on 3.7.2



https://i.postimg.cc/4yZLWYbZ/Untitled.png (https://postimages.org/)

edit: anything newer than Avisynth+ 3.7.3 test 7 (20230223) will give same error

[kedautinh12]

I'm also having this issue. I'd love to enjoy 3.7.3 but haven't been able to resolve this so I'm stuck on 3.7.2

Cause you don't update to 3.7.3 beta version. You just use the 3.7.3 stable version

[Dogway]

What's currently the recommended audio loading filter in avisynth? I generally use ffms2 but apparently bestaudiosource is better/less bugs? The thing is bestaudiosource hasn't been updated in 3 years.

[Emulgator]

LWLibavAudioSource here, occasionally FFAudioSource

[ravewulf]

Be careful with LWLibavAudioSource and the Dolby codecs as LWLibavAudioSource applies DRC by default. drc_scale needs to be manually set to 0 to avoid it

https://github.com/HomeOfAviSynthPlusEvolution/L-SMASH-Works/issues/5

[Julek]

The thing is bestaudiosource hasn't been updated in 3 years.
A big update is coming.

And the current repo is this one:
https://github.com/vapoursynth/bestsource

[Kurt.noise]

Hi,

I would like to test the channelmask internal functions from the latest stable release (3.7.3) using this script :

LoadPlugin("C:\Users\LionelDUCHATEAU\Downloads\megui_git\megui\bin\x64\Debug\tools\lsmash\LSMASHSource.dll")
LWLibavAudioSource("C:\Users\LionelDUCHATEAU\Downloads\AC3 Samples\8w3D.eac3", drc_scale=0)
AudioDubEx(BlankClip(length=Int(1000*AudioLengthF(last)/Audiorate(last)), width=720, height=720, fps=25), last)
m=IsChannelMaskKnown()
g=GetChannelMask()
s=SetChannelMask(true,g)

Info()
Subtitle(
\ "\nAudioLength = " + String(AudioLength)
\ + "\nAudioLengthS = '" + AudioLengthS + "'"
\ + "\nAudioLengthF = " + String(AudioLengthF)
\ + "\nAudioLengthLo= " + String(AudioLengthLo)
\ + "\nAudioLengthHi= " + String(AudioLengthHi)
\ + "\nIsChannelMaskKnown= " + String(m)
\ + "\nGetChannelMask= " + String(g)
\ + "\nSetChannelMask= " + String(s)
\ , font="courier", text_color=$ffffff, size=32, align=4, lsp=0)

but String(s) doesn't return something whereas I get information from Info(). Did I miss something here ?

[rgr]

I'm pretty sure it's the "frozen as ice" issue with avstp.dll I faced a long time ago and that Ferenc fixed.

See here: https://github.com/pinterf/mvtools/issues/46

Either update to the new avstp.dll https://github.com/pinterf/AVSTP/releases or get rid of the one you have in the plugins folder.
I'm pretty sure that it will solve the issue ;)

I updated a few things in the last 2 months (including AviSynth and AVSTP) and haven't had any freezes since. So something helped :)

[FranceBB]

I updated a few things in the last 2 months (including AviSynth and AVSTP) and haven't had any freezes since. So something helped :)

Yep, Ferenc fixed it in avstp.
I mean, of course he did, he's amazing.

[Emulgator]

Be careful with LWLibavAudioSource and the Dolby codecs as LWLibavAudioSource applies DRC by default. drc_scale needs to be manually set to 0 to avoid it.
Thanks ravewulf. Comes in handy, just developing a snippet where this can be used:
https://forum.doom9.org/showthread.php?t=167435&page=80

[pinterf]

Hi,

I would like to test the channelmask internal functions from the latest stable release (3.7.3) using this script :

LoadPlugin("C:\Users\LionelDUCHATEAU\Downloads\megui_git\megui\bin\x64\Debug\tools\lsmash\LSMASHSource.dll")
LWLibavAudioSource("C:\Users\LionelDUCHATEAU\Downloads\AC3 Samples\8w3D.eac3", drc_scale=0)
AudioDubEx(BlankClip(length=Int(1000*AudioLengthF(last)/Audiorate(last)), width=720, height=720, fps=25), last)
m=IsChannelMaskKnown()
g=GetChannelMask()
s=SetChannelMask(true,g)

Info()
Subtitle(
\ "\nAudioLength = " + String(AudioLength)
\ + "\nAudioLengthS = '" + AudioLengthS + "'"
\ + "\nAudioLengthF = " + String(AudioLengthF)
\ + "\nAudioLengthLo= " + String(AudioLengthLo)
\ + "\nAudioLengthHi= " + String(AudioLengthHi)
\ + "\nIsChannelMaskKnown= " + String(m)
\ + "\nGetChannelMask= " + String(g)
\ + "\nSetChannelMask= " + String(s)
\ , font="courier", text_color=$ffffff, size=32, align=4, lsp=0)

but String(s) doesn't return something whereas I get information from Info(). Did I miss something here ?
SetChannelMask returns the clip itself, you cannot stringify it.

If you'd like to get the friendly name of the channel mask constant combination, it's not possible. It's used by Info, so it is only an internal function.

Docs.
https://avisynthplus.readthedocs.io/en/latest/avisynthdoc/corefilters/setchannelmask.html

[pinterf]

Yep, Ferenc fixed it in avstp.
I mean, of course he did, he's amazing.
Your welcome, yes, the culprit was possibly avstp.

[pinterf]

What's happening in the background:
There are some difficult, though very specific phenomenons under investigation, such as
- why memory consumption grows when someone manually singlesteps forward 40 then backward 10 frames in Avspmod
- is it theoretically possible to eliminate the ever-growing behavior of Avisynth's string heap, which can be vry agressive if a runtime function (ScriptClip) contains a lot of string operation
They are both annoying ones but still, interesting challenges to solve.

Then there are some ideas from qyot27 about supporting/emulating frame property inheritance / setting for old plugins and filters which possibly would never get rebuilt.

- implemented 'continue' and 'break' for control loops, after Asd-g's request. (no build from me yet)
More info at
https://avisynthplus.readthedocs.io/en/latest/avisynthdoc/syntax/syntax_control_structures.html#using-break-and-continue-in-loops

[FranceBB]

- implemented 'continue' and 'break' for control loops

uuuuuuuuuuh very very nice! :D

[wonkey_monkey]

- why memory consumption grows when someone manually singlesteps forward 40 then backward 10 frames in Avspmod

Just Avspmod, not VirtualDub? I have my own viewer I can test against if there's a particular clip that has this behaviour.

[pinterf]

Just Avspmod, not VirtualDub? I have my own viewer I can test against if there's a particular clip that has this behaviour.
Probably the situation is the same with VirtualDub and with any tool that can request arbitrary frames.

[pinterf]

Probably the situation is the same with VirtualDub and with any tool that can request arbitrary frames.
Not necessarily. I don't know what AvsPmod is doing.

I put several debug lines to see what the real frame sequence that must be served by Avisynth (avs_get_frame C interface call). In order to see, which specific frame sequence fools Avisynth prefetcher prediction logic. (I supposed that when I do a backward single-step in Avspmod, Avisynth prefetcher will recognize the new - now negative - pattern, and the Prefetcher will lock into the new delta=-1 prefetching schema.)

And it seems that AvsPMod is requesting the same frame numbers multiple times during its display, even in arbitrary (not one-by-one) order

I see multiple clips, two Prefetch statements, which made the debugging a bit harder :), maybe they are different clips, one before the resizing - the actual unaltered clip - and one after the display conversion?

E.g. sometimes, when I press the "back-reverse", it skips back by 2 frames on one of its script instance, which ends with Prefetch(1,1).
And only by 1 frames with another (?) script instance which ends with the Prefetch(4) - the last line in the script.

@gispos, can you help me with it? The actual issue on github is:
https://github.com/AviSynth/AviSynthPlus/issues/379#issuecomment-2048852993
after a doom9 report:
https://forum.doom9.org/showthread.php?p=1995403#post1995403
Thank you

[tormento]

Do we still need to declare SetMemoryMax?

I mean, almost 70% of the computers are now 64-bit with gazillions of GBs of RAM.

Is it of any use? Can't we simply let modern Windows to manage memory?

[LigH]

You can probably omit it quite safely. Avisynth uses a convenient default.

[Boulder]

The default is quite conservative and is sometimes not enough if you work on 4K sources with heavy filtering and use multithreading. There's no real harm setting the max higher if you have the memory available, Avisynth will use what it needs and not all of it automatically.

[tormento]

Perhaps I didn't explain well.

I know I can set it as I want but, seen VaporSynth memory use, why can't AVS+ go that way too and send Setmemorymax to ancient memories?

[tormento]

The default is quite conservative and is sometimes not enough if you work on 4K sources with heavy filtering and use multithreading.
I can't find my post but, some years ago, I did some tests where I saw performance decrease with Setmemorymax increase.

[FranceBB]

I know I can set it as I want but, seen VaporSynth memory use, why can't AVS+ go that way too and send Setmemorymax to ancient memories?

It's already ancient memory.
There's no need to set it, Avisynth will manage that automatically.
By default, in modern x64 systems, it will use 4GB of RAM which is plenty.
Remember that this is the memory of the cache, in other words the memory used to store frames without having to fetch them again, NOT the RAM used by filters in general. This means that even if you were to set SetMemoryMax(512) but you were heavily filtering a UHD content, Avisynth would still use way more than 512MB of RAM. Remember that the cache is there only so that if you have temporal filters and encoders that need to access frames in a non linear fashion, Avisynth won't have to compute them every single time it moves forward and backward and forward and backward etc. The bigger the cache the more frames are gonna be stored, but storing too many frames can be detrimental as it's gonna use RAM that could otherwise be used by - let's say - the encoder, which is why the cache size is calculated on the fly on the basis of the available RAM and it never exceeds 4GB.

Assuming RGB48 (which is overkill anyway), we have:

3840×2160×48 = 398131200 bits = 0.0498 gigabyte

That's just for one frame, so at the current default Avisynth can cache more than 80 frames. Obviously working with RGB48 UHD is a bit unrealistic, but if we work in YUV that number increases and it even goes much further if we work in 8bit and at lower resolutions.
My point is that it's already implemented correctly and I don't see a reason to change that. If anything, having the SetMemoryMax() is useful if you wanna reduce the cache. ;)

[DTL]

It looks you not understand how frame-based AVS+ MT is working.

To make things as fast as possible (to minimize CPU stall on threads sync) it looks AVS+ simply put several frames cache around _each_ filter in the filtergraph and in _each_ logical thread. So any time system have some free logical CPU cores it can load it with some useful computing.

But it cause awful RAM consuming in any 'complex' scripts of a several filters in a chain.

Total RAM in a cache is about NumFrames_in_Prefetch_x_NumFilters_x_NumThreads_x_FrameSize.

[Boulder]

Regarding the 4GB, which is unfortunately not enough in many cases: https://forum.doom9.org/showthread.php?p=1913375#post1913375

[tormento]

Regarding the 4GB
AVSMeter can provide realistic results?

(Where on hell has Groucho2004 gone?)

Secondo question: SetCacheMode(1) is a good choice?

[FranceBB]

Regarding the 4GB, which is unfortunately not enough in many cases: https://forum.doom9.org/showthread.php?p=1913375#post1913375

Another interesting thing about prefetch, I see!



Total RAM in a cache is about NumFrames_in_Prefetch_x_NumFilters_x_NumThreads_x_FrameSize.

I see! But to clarify I don't actually use Prefetch, like ever, I'm on the good old concept of letting plugin developers handle multithreading, so my scripts don't have it.
For instance, my scripts are like:

video=LWLibavVideoSource("video.mxf")
audio=LWLibavAudioSource("audio.mxf")
AudioDub(video, audio)

ConvertBits(16)

ConvertYUVtoXYZ(Color=0, OutputMode=1, HDRMode=0, fullrange=false)

ConvertXYZ_Reinhard_HDRtoSDR(exposure_X=2.5, contrast_X=0.9)

ConvertXYZtoYUV(pColor=0)


without prefetch. In the example above, the filters would create their own threadpool which is how I think it's intended to be. Call me traditionalist but I see Prefetch() as an evolution of the old MT Modes from 2009 (https://forum.doom9.org/showthread.php?t=148782) and in my head they should only be used if the filter you're calling is old and single-threaded only and of course at your own risk as it might misbehave, especially if it filters temporally.


This however brings me to a different question, then.
Is there an advantage in raising the SetMemoryMax() value from 4GB to something higher IF I don't use Prefetch()?

[DTL]

If you not use Prefetch() it mean AVS+ will run all filtergraph with single threaded cache mode ? So the equation for total cache size is about
NumFrames_in_Prefetch_x_NumFilters_x_FrameSize.

Default num frames in AVS+ cache (for each filter) is at least several ? (2..3 to 10 ? or num of physical cores ?).

"Is there an advantage in raising the SetMemoryMax() value from 4GB to something higher IF I don't use Prefetch()?"

It may depends on workflow. And easy tested between 4 GB and all RAM avaialble. The idea of SetMemoryMax() may be complex hint to AVS memory manager like to save from swapping of too much allocated cache pages attempt to lower number of cached frames if total process RAM allocation approaches the current RAM limit.

Because with increasing number of frames in the cache user may got some performance boost (depending of filters used and many more effects) setting too low MemoryMax may cause decreasing of performance in some use cases. It is better to check with exact end user workflow.

[Boulder]

AVSMeter can provide realistic results?

(Where on hell has Groucho2004 gone?)

Secondo question: SetCacheMode(1) is a good choice?

Windows Task Manager shows the same information. With heavy prefetch settings like threads=32, frames=12 on my 5950X, a 4K source with MVTools based filtering and downscaling with avsresize's functionalities, outputting 16-bit data to x265 can take 20GB of memory. The amount of frames to prefetch is the key to memory usage. That's why it is important to restrict it if you have a lot of cores, the default is way too much (would be 64 frames in this case :eek:)

I've found out that mode 1 is better.

[Boulder]

This however brings me to a different question, then.
Is there an advantage in raising the SetMemoryMax() value from 4GB to something higher IF I don't use Prefetch()?

I would say probably not. I've been running my chunked encoding tool and the memory usage is usually max 3GB per avs2yuv64 process with my standard denoising and resizing functions. Though it's with 10-bit output, I've not tested if 16-bit output changes things that much to go over the 4GB limit.

[tormento]

My findings.

Basic script

SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("M:\In\2013 GIT - Arise ~864p Dynit\2013_06 Ghost pain.dgi")
Comptest24(1)
Convertbits(16)
libplacebo_Resample(1536,864,filter="ewa_lanczossharp")
SMDegrain (tr=3, thSAD=300, refinemotion=false, contrasharp=false, PreFilter=7, plane=4, chroma=true, mode="MDegrain")
libplacebo_Deband(iterations=3, temporal=true)
fmtc_bitdepth (bits=10,dmode=7)
Prefetch(3)

AVSMeter 3.0.9.0 (x64), (c) Groucho2004, 2012-2021
AviSynth+ 3.7.3 (r4071, 3.7, x86_64) (3.7.3.0)

Number of frames: 1680
Length (hh:mm:ss.ms): 00:01:10.070
Frame width: 1536
Frame height: 864
Framerate: 23.976 (24000/1001)
Colorspace: YUV420P10

Frames processed: 1680 (0 - 1679)
FPS (min | max | average): 1.202 | 116279 | 11.54
Process memory usage (max): 1990 MiB
Thread count: 45
CPU usage (average): 35.7%

GPU usage (average): 15%
VPU usage (average): 8%
GPU memory usage: 1496 MiB
GPU Power Consumption (average): 41.6 W

x265 slow: encoded 1680 frames in 297.25s (5.65 fps), 1134.83 kb/s, Avg QP:23.77

Adding on top:

SetMemoryMax()

FPS (min | max | average): 1.589 | 55866 | 11.33
Process memory usage (max): 1947 MiB
Thread count: 45
CPU usage (average): 35.6%

x265 slow: encoded 1680 frames in 293.64s (5.72 fps), 1138.50 kb/s, Avg QP:23.70

Adding on top:

SetCacheMode(1)

FPS (min | max | average): 0.340 | 178572 | 11.52
Process memory usage (max): 1849 MiB
Thread count: 45
CPU usage (average): 35.7%

x265 slow: encoded 1680 frames in 297.86s (5.64 fps), 1137.39 kb/s, Avg QP:23.72

Adding on top:

SetMemoryMax()
SetCacheMode(1)

FPS (min | max | average): 0.338 | 133333 | 11.42
Process memory usage (max): 1846 MiB
Thread count: 45
CPU usage (average): 35.6%

x265 slow: encoded 1680 frames in 284.14s (5.91 fps), 1136.06 kb/s, Avg QP:23.72

I see some discrepancies between avs+ only and real world scenario with x265 encoding but, perhaps, is just my old machine.

[DTL]

"SetMemoryMax() "

I think it requires memory size (in MBytes ?). Empty params call do not change current AVS memory max.

SetMemoryMax(amount)
Sets the maximum memory that AviSynth uses (in MB) to the value of amount. Setting to zero just returns the current Memory Max value. In the 2.5 series the default Memory Max value is 25% of the free physical memory, with a minimum of 16MB.
The default Memory Max is also limited to 512MB.
AVS+In Avisynth+ this limit for default Memory Max is 1024MB for 32 bits and 4096MB on the x64 version
DefaultMemoryMax = minimum(physical_memory / 4, secondary_memory_max_limit)

This really working:

ColorBars()

mem=SetMemoryMax(0)
SubTitle(Format("old default max={mem}!"))


SetMemoryMax(5000)
mem=SetMemoryMax(0)
SubTitle(Format("new max={mem}!"), align=4)


First call returns old memory max value (4074 at me) and second call sets to 5000 MB.

So you can first check your current system default memory max and check some significant changes like set to 10 times lower and about 70..80% of current RAM installed.

[guest]

Where did this come from ??

AviSynth+ 3.7.3 (r4071, 3.7, x86_64) (3.7.3.0)

[DTL]

From documentation at http://avisynth.nl/index.php/Internal_functions

http://avisynth.nl/index.php/Internal_functions#SetMemoryMax

[guest]

From documentation at http://avisynth.nl/index.php/Internal_functions

http://avisynth.nl/index.php/Internal_functions#SetMemoryMax

Sorry DTL, you misunderstood :(

Where did that build of Avisynth come from, (r4071) In tormento's post.

But :thanks:

Found it....https://gitlab.com/uvz/AviSynthPlus-Builds

[tormento]

"SetMemoryMax() "

I think it requires memory size (in MBytes ?). Empty params call do not change current AVS memory max
I look at numbers :)

Something happens and is not what I expected.

Some years ago I did benchmarks here with various amount of memory and that () lead to the best results, even against higher ones.

[hello_hello]

Is it intended for the Convert functions to change the chroma location when they're not actually changing the color format?

For example, if you have a YUV420 source with the chroma location in frame properties as "top_left", and you add ConvertToYUV420() to the script, the color format doesn't change but the chroma location changes to "left".

I know "left" is the default output chroma location, and it works as the Avisynth wiki says it does, but before I noticed the chroma location had changed and checked, I expected it to remain the same unless the color format had changed.

I can't decide if changing the location when there's no conversion taking place is a good idea. Or maybe it doesn't matter much....

Cheers.

[FranceBB]

I'm back with some examples.
After the last discussion about SetMemoryMax() I did some more tests and this time I used a real script from a real use case scenario (although I trimmed the first 1000 frames only as I didn't want to let it filter 40 minutes worth of documentary for nothing).


Source: DNX HQX UHD 4:2:2 12bit 728 Mbit/s BT709 SDR

Video
ID : 2
Format : VC-3
Commercial name : DNxHR HQX
Format version : Version 3
Format profile : RI@HQX
Format settings, wrapping mode : Frame
Codec ID : 0D01030102110100-0401020271250000
Duration : 40 min 53 s
Bit rate mode : Constant
Bit rate : 728 Mb/s
Width : 3 840 pixels
Height : 2 160 pixels
Display aspect ratio : 16:9
Active Format Description : Full frame 16:9 image
Frame rate : 25.000 FPS
Color space : YUV
Chroma subsampling : 4:2:2
Bit depth : 12 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 3.512
Stream size : 208 GiB (100%)
Color range : Limited
Color primaries : BT.709
Transfer characteristics : BT.709
Matrix coefficients : BT.709



AVS Script:


video=LWLibavVideoSource("PARTE 1.mxf")
FL=WAVSource("Parte 1 Mix OnAir.L.wav")
FR=WAVSource("Parte 1 Mix OnAir.R.wav")
CC=WAVSource("Parte 1 Mix OnAir.C.wav")
LFE=WAVSource("Parte 1 Mix OnAir.LFE.wav")
LS=WAVSource("Parte 1 Mix OnAir.Ls.wav")
RS=WAVSource("Parte 1 Mix OnAir.Rs.wav")

Dolby=MergeChannels(FL, FR, CC, LFE, LS, RS)

Stereo=WAVSource("Parte 1 MIX Lt - Rt.wav")

Mute=BlankClip(length=61335, fps=25, audio_rate=48000, channels=2)

audio=MergeChannels(Stereo, Mute, Dolby, Mute, Mute, Mute)

AudioDub(video, audio)


ConvertBits(16)

z_ConvertFormat(pixel_type="RGBP16", colorspace_op="709:709:709:limited=>rgb:709:709:full", resample_filter_uv="spline64", dither_type="error_diffusion", use_props=0)

Cube("C:\Program Files (x86)\AviSynth+\LUTs\5a_BT709_HLG_UPCONVERT_DISPLAY_mode.cube", fullrange=1, interp=1)

z_ConvertFormat(pixel_type="YUV422P10", colorspace_op="rgb:std-b67:2020:full=>2020:std-b67:2020:limited", resample_filter_uv="spline64", dither_type="error_diffusion", use_props=0)

Limiter(min_luma=64, max_luma=940, min_chroma=64, max_chroma=960)

trim(0, 1000)




AVS Meter Test:


AvsMeter64.exe "Test.avs"

pause



Hardware:

CPU: Intel Xeon Gold 6238R 2.20GHz x2 (56c/112th)
RAM: 32 x 4 DDR4 1460 MHz (128 GB)
Motherboard: HPE ProLiant XL420 Gen10
GPU: HP Matrox Matrox G200eh3 32MB
Storage 1: 447GB Hitachi
Storage2: 4471GB Hitachi
NIC: HP 562SFP+ 10 Gigabit

https://i.imgur.com/yoUDN2m.png


Test 1 - no SetMemoryMax() specified (i.e default 4GB)

AVSMeter 3.0.9.0 (x64), (c) Groucho2004, 2012-2021
AviSynth+ 3.7.3 (r4003, 3.7, x86_64) (3.7.3.0)

Number of frames: 1001
Length (hh:mm:ss.ms): 00:00:40.040
Frame width: 3840
Frame height: 2160
Framerate: 25.000 (25/1)
Colorspace: YUV422P10
Audio channels: 16
Audio bits/sample: 24
Audio sample rate: 48000
Audio samples: 1921920

Frames processed: 1001 (0 - 1000)
FPS (min | max | average): 3.484 | 4.231 | 4.084
Process memory usage (max): 808 MiB
Thread count: 73
CPU usage (average): 2.3%

Time (elapsed): 00:04:05.089



Test 2 - SetMemoryMax(25000) specified (i.e 25GB)

AVSMeter 3.0.9.0 (x64), (c) Groucho2004, 2012-2021
AviSynth+ 3.7.3 (r4003, 3.7, x86_64) (3.7.3.0)

Number of frames: 1001
Length (hh:mm:ss.ms): 00:00:40.040
Frame width: 3840
Frame height: 2160
Framerate: 25.000 (25/1)
Colorspace: YUV422P10
Audio channels: 16
Audio bits/sample: 24
Audio sample rate: 48000
Audio samples: 1921920

Frames processed: 1001 (0 - 1000)
FPS (min | max | average): 3.574 | 4.406 | 4.201
Process memory usage (max): 841 MiB
Thread count: 73
CPU usage (average): 2.3%

Time (elapsed): 00:03:58.282



Test 3 - SetMemoryMax(90000) specified (i.e 90GB)

AVSMeter 3.0.9.0 (x64), (c) Groucho2004, 2012-2021
AviSynth+ 3.7.3 (r4003, 3.7, x86_64) (3.7.3.0)

Number of frames: 1001
Length (hh:mm:ss.ms): 00:00:40.040
Frame width: 3840
Frame height: 2160
Framerate: 25.000 (25/1)
Colorspace: YUV422P10
Audio channels: 16
Audio bits/sample: 24
Audio sample rate: 48000
Audio samples: 1921920

Frames processed: 1001 (0 - 1000)
FPS (min | max | average): 3.580 | 4.405 | 4.209
Process memory usage (max): 841 MiB
Thread count: 73
CPU usage (average): 2.3%

Time (elapsed): 00:03:57.812


As we can see from the tests above, even when working on UHD 12bit contents with 16bit precision and doing very heavyweight operations like converting to RGB full range, applying a LUT with tetrahedral interpolation, going back to YUV limited tv range and dithering down to 10bit, the RAM usage never exceeds 4GB, in fact it doesn't matter how much RAM I tell Avisynth to allocate with SetMemoryMax(), it never really uses it anyway and the performances stay the same. This is - of course - without using Prefetch() and I never use it in any of my encodes anyway, so I guess I'm safe to say that I don't really need to specify SetMemoryMax() and I'm pretty much fine with the default value. :)


Here's another test with a different kind of conversion, this time instead of going from BT709 SDR to BT2020 HLG with highlights expansions to 420 nits using the BBC LUT, we're using HDR Tools by Jean Philippe Scotto di Rinaldi to go to BT2020 SDR 100 nits:

AVS Script:

video=LWLibavVideoSource("PARTE 1.mxf")
FL=WAVSource("Parte 1 Mix OnAir.L.wav")
FR=WAVSource("Parte 1 Mix OnAir.R.wav")
CC=WAVSource("Parte 1 Mix OnAir.C.wav")
LFE=WAVSource("Parte 1 Mix OnAir.LFE.wav")
LS=WAVSource("Parte 1 Mix OnAir.Ls.wav")
RS=WAVSource("Parte 1 Mix OnAir.Rs.wav")

Dolby=MergeChannels(FL, FR, CC, LFE, LS, RS)

Stereo=WAVSource("Parte 1 MIX Lt - Rt.wav")

Mute=BlankClip(length=61335, fps=25, audio_rate=48000, channels=2)

audio=MergeChannels(Stereo, Mute, Dolby, Mute, Mute, Mute)

AudioDub(video, audio)


ConvertBits(16)

#BT709 SDR to BT2020 SDR

ConvertYUVtoXYZ()
ConvertXYZtoYUV(Color=1, pColor=2)

ConverttoYUV422(matrix="Rec.2020", interlaced=false)

ConvertBits(bits=10, dither=1)

Limiter(min_luma=64, max_luma=940, min_chroma=64, max_chroma=960)

trim(0, 1000)


Test 1 - no SetMemoryMax() specified (i.e default 4GB)

AVSMeter 3.0.9.0 (x64), (c) Groucho2004, 2012-2021
AviSynth+ 3.7.3 (r4003, 3.7, x86_64) (3.7.3.0)

Number of frames: 1001
Length (hh:mm:ss.ms): 00:00:40.040
Frame width: 3840
Frame height: 2160
Framerate: 25.000 (25/1)
Colorspace: YUV422P10
Audio channels: 16
Audio bits/sample: 24
Audio sample rate: 48000
Audio samples: 1921920

Frames processed: 1001 (0 - 1000)
FPS (min | max | average): 4.594 | 8.925 | 5.896
Process memory usage (max): 1377 MiB
Thread count: 129
CPU usage (average): 16.5%

Time (elapsed): 00:02:49.782


Test 2 - SetMemoryMax(25000) specified (i.e 25GB)

AVSMeter 3.0.9.0 (x64), (c) Groucho2004, 2012-2021
AviSynth+ 3.7.3 (r4003, 3.7, x86_64) (3.7.3.0)

Number of frames: 1001
Length (hh:mm:ss.ms): 00:00:40.040
Frame width: 3840
Frame height: 2160
Framerate: 25.000 (25/1)
Colorspace: YUV422P10
Audio channels: 16
Audio bits/sample: 24
Audio sample rate: 48000
Audio samples: 1921920

Frames processed: 1001 (0 - 1000)
FPS (min | max | average): 4.550 | 8.933 | 5.884
Process memory usage (max): 1377 MiB
Thread count: 129
CPU usage (average): 16.3%

Time (elapsed): 00:02:50.134


Test 3 - SetMemoryMax(90000) specified (i.e 90GB)

AVSMeter 3.0.9.0 (x64), (c) Groucho2004, 2012-2021
AviSynth+ 3.7.3 (r4003, 3.7, x86_64) (3.7.3.0)

Number of frames: 1001
Length (hh:mm:ss.ms): 00:00:40.040
Frame width: 3840
Frame height: 2160
Framerate: 25.000 (25/1)
Colorspace: YUV422P10
Audio channels: 16
Audio bits/sample: 24
Audio sample rate: 48000
Audio samples: 1921920

Frames processed: 1001 (0 - 1000)
FPS (min | max | average): 4.655 | 8.845 | 5.877
Process memory usage (max): 1377 MiB
Thread count: 129
CPU usage (average): 16.0%

Time (elapsed): 00:02:50.337

For those wondering why I'm "happy" with this, please consider that this server is part of a wider farm and would typically run several encodes at the same time, so the CPU is gonna be pegged at 100% anyway. Besides, those tests were made only considering Avisynth, but when x264 is encoding the files in XAVC Intra Class 300 then the usage consumption is of course much higher.

https://i.imgur.com/kFART6b.png

For reference, today the farm encoded 284 files:

https://i.imgur.com/I32nJli.png

[hello_hello]

This is - of course - without using Prefetch() and I never use it in any of my encodes anyway, so I guess I'm safe to say that I don't really need to specify SetMemoryMax() and I'm pretty much fine with the default value.

If you don't add Prefetch, Avisynth+ runs in single threaded mode. Adding Prefetch can make a huge difference to encoding speed, depending on the filtering in the script.

There's a file here that can be saved as an avsi script and auto-loaded by Avisynth+. It tells Avisynth+ the type of multi-threading to use for most of the common plugins. Many plugins register their multi-threading mode with Avisynth+ automatically these days, but they still require you to add Prefetch to the script to activate multi-threading.

https://publishwith.me/ep/pad/view/ro.rDkwcdWn4k9/latest

More info
http://avisynth.nl/index.php/AviSynth+#Help_filling_MT_modes

I assume for plugins with "internal" multi-threading it's a different thing to enabling Avisynth's own multi-threading. Ideally for some plugins, such as the ones in the JPSDR plugins pack, you should use the plugin arguments to specify how many threads Avisynth+ is using when you've added Prefetch to the script. And sometimes it's better to disable a plugin's multi-threading. Apparently it's not a good idea to use avstp.dll for multi-threading when Avisynth's multi-threading is enabled. The script I linked to above also tells avstp.dll to run in single threaded mode, assuming it's loaded.

[cretindesalpes]

Yep, Ferenc fixed it in avstp.
I mean, of course he did, he's amazing.

Sorry I’m a bit late to the party. I read the Ferenc debugging report (https://github.com/pinterf/AVSTP/issues/1). Very interesting, nice catch! I saw that an assert occured in:

++ loop_cnt;
if (loop_cnt >= max_loop)
{
// This could indicate that the queue is:
// - corrupted
// - or in heavy contention
assert (false);
return nullptr;
}

Obviously returning 0 is not expected at all by the calling code and makes everything fail. But have you tried to increase max_loop? The value in the code is totally arbitrary, I haven’t done any statistics on the maximum loop_cnt I could encounter in a heavy use, but maybe we are sometimes close to the limit (contention case)? (edit: the GRAME paper doesn’t even have this check in the presented algorithm, I probably added it during development).

Anyway, I love the irony of the fix: enclosing a “lock-free” code in a std::lock_guard :D
BTW this could be achieved more simply using a trivial std::list or std::deque for storage with guarded access instead of the complicated lock-free procedure. But I’m glad you fixed it!

[FranceBB]

Currently, when a Convertto function is called, it uses the matrix from the frame properties.
This is ok, however it only really works when the file is right and the indexer populates the frame properties correctly, which is not always the case.
Here's a sample file: https://we.tl/t-aJB9L84pmw (link valid for 7 days)
(keep in mind that the file ain't important, it's just an example to make a point).

When it's being indexed by FFVideoSource() it shows the following frame properties:

https://i.imgur.com/VFFvwDd.png

Clearly Matrix(3) doesn't mean anything, but you can bet everything you want that ConverttoRGB24() is gonna try to use that and fail, in fact it fails with "Unknown matrix".

Of course one could always nuke frame properties with propClearAll() before using ConverttoRGB24() and call it a day, however that's not exactly ideal.

What I'm proposing here is to still allow the Convertto functions to use the matrix populated by the frame properties as it is today, but, if it's populated automatically and not passed by the user explicitly, to fallback to a safe default instead of throwing an error if what is passed through is garbage.

[ENunn]

I can't reproduce the problem with last test 14 (r4066), what is your installed version?
If it is the same version please upload the .avs used and the source (or a sample)

Sorry for the months late reply.

I just tried it again, and I'm still having the issue. It's not happening with every script, but some.

Source video (https://mega.nz/file/X19HRZQT#cB05_0cj0HJjEttX3Pxrd1QSsUmD1R65frvtNkF2scg)
Source audio (https://mega.nz/file/3scj3RTJ#sl-pV9hRn_9VTjO-IoYEiqjnyn6Wic2NHtvTHH74Mbs)

Script:
v = lwlibavvideoSource("f:\virtualdub\tape transfers\opening to swing time 1996 vhs - edit.mkv", fpsnum=30000,fpsden=1001)
a = lwlibavaudiosource("d:\recordings\opening to swing time 1996 vhs - edit.flac")
audiodub(v,a)
#delayaudio(-.150)
assumetff().converttoyuv422(matrix="rec601", interlaced=true).convertbits(10)
#Crop(8, 4, -24, -6_
Levels(50, 1,920, 0, 1020, coring=false,dither=true).tweak(bright=0, cont=1.00, hue=-0, sat=1.00, coring=false, dither=true).convertbits(8)
#turnRight().Histogram().TurnLeft()
normalize(0.8912)
Trim(415, 5286)
prefetch(8)

Example (https://mega.nz/file/3tkR3DrQ#pIYwhumwtu2GDiwn9R1n2mncaYzNpR3f3VoFsPMMr4c) WARNING: LOUD!!!

[tebasuna51]

Yes, there are a problem and I can't understand where.

Played your script in VirtualDub2 sound noise and distorted audio.

Without the normalize() play fine, but also without the normalize play noise in mpc_hc.

Tested also the FFAudioSource and BSAudioSource decoders, and only the audio without video, normalize, etc., always noise and distort.

Maybe something related with AviSynth flac decoders?
Decoded with ffmpeg or flac seems fine.

[LigH]

There is FLAC with 24 bit resolution. That might be unusual for an Avisynth audio decoder and sound wrong when it does not report the correct sample layout.

Can you check technical details of your audio source, e.g. with MediaInfo or FLAC tools?

[ENunn]

Yes, there are a problem and I can't understand where.

Played your script in VirtualDub2 sound noise and distorted audio.

Without the normalize() play fine, but also without the normalize play noise in mpc_hc.

Tested also the FFAudioSource and BSAudioSource decoders, and only the audio without video, normalize, etc., always noise and distort.

Maybe something related with AviSynth flac decoders?
Decoded with ffmpeg or flac seems fine.

I had the same issue with an ac3 file, but as I said before, it seems to happen on some files, but not all.

There is FLAC with 24 bit resolution. That might be unusual for an Avisynth audio decoder and sound wrong when it does not report the correct sample layout.

Can you check technical details of your audio source, e.g. with MediaInfo or FLAC tools?
General
Complete name : D:\recordings\opening to swing time 1996 vhs - edit.flac
Format : FLAC
Format/Info : Free Lossless Audio Codec
File size : 15.9 MiB
Duration : 3 min 0 s
Overall bit rate mode : Variable
Overall bit rate : 740 kb/s

Audio
Format : FLAC
Format/Info : Free Lossless Audio Codec
Duration : 3 min 0 s
Bit rate mode : Variable
Bit rate : 740 kb/s
Channel(s) : 2 channels
Channel layout : L R
Sampling rate : 48.0 kHz
Bit depth : 16 bits
Compression mode : Lossless
Stream size : 15.9 MiB (100%)
Writing library : libFLAC 1.3.2 (2017-01-01)
MD5 of the unencoded content : 3C5F271BC902D1430FDC4F3D062F4A21


This was a FLAC created by Adobe Audition. I usually do some dehumming and denoising. I imported the raw audio, also in FLAC (but this time converted with ffmpeg), and the same issue happens. I did try converting it to PCM with ffmpeg as well, and the issue persists. This doesn't happen with r4003.

[qyot27]

I can't reproduce it here. Samples as posted, script as posted. LSMASHSource built on May 1, freshly built AviSynth+ r4076 plugnew, Ubuntu 24.04. There's no noise or distortion.

[ENunn]

Just to be safe, I updated LSMASHWorks, and the issue persists.