AviSynth+ thread Vol.2

[pinterf]

Latest test build (https://github.com/pinterf/AviSynthPlus/releases/tag/v3.7.6pre-r4554)
Avisynth+ v3.7.6pre-r4554 test build (20260307)


20260307 3.7.5.r4554 (pre 3.7.6)
--------------------------------
- Fix: Working 'quality' parameter for 10-14 bits sources in YUV<->RGB conversions (was: blackness)

20260306 3.7.5.r4551 (pre 3.7.6)
--------------------------------
- Reenable YUY2 in planar target conversions as source (disabled in r4549)
- ConvertToY also gets bits and quality parameters, latter is used when source is RGB.

20260305 3.7.5.r4549 (pre 3.7.6)
--------------------------------
- Fix: memory leak in Subframe/MakePropertyWritable after static-frame sources (ColorBars, BlankClip)
- any->YUV conversions (See https://avisynthplus.readthedocs.io/en/latest/avisynthdoc/corefilters/convert.html`.
- accept bits and quality parameters, similar to ConvertToPlanarRGB
- the legacy 8-bit-named functions (``ConvertToYV12``, ``ConvertToYV16``,
``ConvertToYV24``) allowing high-depth sources.
- "ConvertToYUY2": rewritten to route all conversions through YV16.
- "ConvertToYUY2": ``ChromaOutPlacement`` parameter added (was missing, present in ConvertToYV16).
- "ConvertBackToYUY2": kept for backward compatibility; now forwards to ``ConvertToYUY2``. The pre-2.5 left-pixel-only chroma hack is no longer
needed or applied; the YV16 lossless repack path avoids chroma resampling loss entirely for roundtrip workflows.
- 8 bit packed RGB formats are converted to planar RGB before 444 conversion.
Utilizing the optimized planar RGB infrastructure.
- Fix: "ConvertToYUY2" / "ConvertToYV12": progressive YV12<->YUY2 conversion (use generic YV16/YV12 path)
- Fix: "ConvertToYUY2": ``_ChromaLocation``, ``_Matrix`` and ``_ColorRange``
frame properties were not read from YV12 source frames and not written to YUY2 output frames
in the old legacy direct conversion path.
- Fix: "ConvertToYUY2": SSE2 interlaced upsampling used wrong weighting direction, differing from the C reference implementation.
- Update https://avisynthplus.readthedocs.io/en/latest/avisynthdoc/corefilters/convert.html`
- matrix syntax
- ConvertToYUY2
- ``bits`` and ``quality`` parameters
- Update Sampling https://avisynthplus.readthedocs.io/en/latest/avisynthdoc/advancedtopics/sampling.html with historical content notes
on legacy ``YUY2`` handling.

20260220 3.7.5.r4529 (pre 3.7.6)
--------------------------------
- Fix Colorbars inaccurate 10+ bit, by using ground truth linear RGB, similarly to ColorBarsHD.
- Full refactoring.
- update doc: https://avisynthplus.readthedocs.io/en/latest/avisynthdoc/corefilters/colorbars.html
- Histogram "color" and "color2" (Vectorscope modes)
- full refactoring.
- Drawing is now matrix and color range aware. target positions (75%) +-I and +Q.
- add individual overlay options:
added ``matrix``, ``graticule``, ``targets``, ``axes``, ``iq``, ``iq_lines``, ``circle``
parameters
- Fix: copy alpha from clip, initialize alpha to zero in the histogram area.
- Accurate pixel positioning and scaling to the active histogram area,
limited/full range aware.
- update doc: https://avisynthplus.readthedocs.io/en/latest/avisynthdoc/corefilters/histogram.html

[tormento]

- Fix: Working 'quality' parameter for 10-14 bits sources in YUV<->RGB conversions (was: blackness)
Would you please explain this more in detail?

I can find no reference about that feature/parameter.

I've found fmtconv to be the most precise among the conversion plugins. If AVS+ internals could become so precise and even fast, they would be welcomed ones.

[jpsdr]

@pinterf
Hi.
What is the resamplers status ?
Are the new functions (AVX2 & AVX512) Ok, and can i begin to try to implement them in my ResamplerMT, or are they still in testing mode ?
Just to try to avoid to spend hours for nothing if i have to redo everything later. By later i mean shortly, not if something in a few months append and a fix/change is needed.

[DTL]

Most important update on resamplers for permutex-based functions still not ported to AVS+ is yet another pre-processing of the coeffs in the resampling program. It looks universal for all H to V permute-based transposition functions. See this commit https://github.com/DTL2020/AviSynthPlus/commit/a2604a2fcbb0589e578640a376901c03dc9c9f7e

Also some simple data locality optimization at the 2d resizing of multi-planes (typical colour) formats - process each plane in 2 resizers in a sequence instead of processing all planes in first dimension at first and all planes in second dimension at next stage https://github.com/DTL2020/AviSynthPlus/commit/810f91ba468601316f8ddf89adee3c9821eb9bfd . This allows output of the first resize to be hot in cache if possible to be read by second stage.

[jpsdr]

I have a question about:
cache_size_L2 = env->GetEnvProperty(AEP_CACHESIZE_L2);
What is the result if called in an avisynth version not supporting AEP_CACHESIZE_L2 ?
In that case, what default value should be used ?

Edit:
Looking at resampler_h_detect_optimal_scanline it seems that if unsupported return 0, it's Ok.

@DTL
I'm in a special lucking rare case, work is for now quiet, so i would say that i have time for begin to check how to implement things.
But in the opposite, i don't have time for now when back home for testing. I've pushed for now in my ResampleMT the SSE and AVX2 update. As it was only for float cases, i was in a way very easy, as i don't have to care about my range and YUY2 mode parameters ;).
But no check for now. So, if on your side you have time to build and test, be my guest... :D
If not, no hard feeling, of course !

Edit2:
Forgot to ask... Why there is resample_avx512.cpp, resample_avx512.h but also resample_avx512.hpp and resample_avx512b.cpp and not just the first two ?

Edit3:
@DTL
You and pinterf made a lot of work, thanks.
You talked about a commit not yet in avs+, it means that your repo is up to date to current avs+, with something more you made, am i correct ?
Is what you made "final", or is there some benchmarks to make to finalize like pinterf done ?
Do you think just taking the "benchmark winners" and removing all the unused functions would be enough ? It will reduce the work for me to implement in ResampleMT.

[DTL]

"Why there is resample_avx512.cpp, resample_avx512.h but also resample_avx512.hpp "

As I remember this was a special hack by pinterf to support many compilers (and many OSes/architectures ?) builds with AVX512. If you build with only MSVS it is not required. This makes program text more complex.

[jpsdr]

Thanks, and i'll keep only the "winners" functions. Not realy thrilled to spend hours for implementing dead code... :(

[DTL]

" it means that your repo is up to date to current avs+, with something more you made, am i correct ?"

It is based on the 4439 commit to main AVS+ branch and newer commits are not included.

" just taking the "benchmark winners" and removing all the unused functions would be enough ? "

There are many function versions and too few real hardware resources for testing. I can only access old Xeon about mid 201x years and pinterf can test at the 'slow' consumer grade intel-11 CPU. So the real winners at the new AVX512 and CPU cache architectures (like AMD Zen5 and new intels) may be different.

The vertical resizers are few and universal (for any filter size and any resample ratio) and horizontals are much more complex and many are limited to only small filter sizes and generally upsample (and very small downsample) ratios. One use case of the 2x upsampling and /2 downsampling with short kernel BicubicResize in AVS+ core is to work in ConvertXXX for chroma upsize/downsize at the different chroma subsampling conversions.

There still no generic AVX512 H-resize function to support all filter sizes and all resample ratios. Some attempts to redesign AVX2 H-resize generic to AVX512 were not good in performance (may be also at old/slow AVX512 CPUs ?).

Also performance of the V-stripe scan H-resizers (without coeffs reloading and keeping in register file) is only good if in/out plane size fits in CPU caches. And this memory scan pattern (for store) is very non-friendly with main host RAM (SDRAM). Some attempts in partial V/H memory access pattern interleaving (like an H-stripe of about L2 cache size filled with partial V-stripes with still less coeffs reload per frame and forcing ready H-stripe to flush to host RAM as linear write stream) may be not complete winners at all use cases (only a few tested).

The real top-performance H-resizers as a special plugin may be designed like fftw software with performance measuring at current host with current task options of different implementations and selecting best. The processing time of the long footage is very long and time addition for small performance testing may be not visible at the full processing of the footage. Or you can make manual tuning options for users to try different implementations (and other performance-tuning options of each implementation like number of rows in a stripe) to select the best if the user cares.

Currently selected functions to AVS core may be only 'local winners' but fail with some not tested possible use cases (like very large frame sizes). Even with some automation script designed by pinterf it is still a long task to test very different use cases of the resizers.

Performance may also significantly depend on the threads number and host memory performance (frequency, bus width, memory channels number). In the AVS+ resizer the single function (single CPU core/thread) processes the whole plane and in ResampleMT several threads can process parts of a plane and this also may change the total performance and need testing again.

" resample_avx512b.cpp"

Because AVS+ core must support many compilers and also many CPU cores (with different AVX512 instructions sets - separated to AVX512_BASE and AVX512_FAST now in cpuid.h):
1. Functions with AVX512 instructions up to the 'base' set (old AVX512) separated in avx512b.cpp file.
2. Functions with all AVX512 instructions up to AVX512_FAST (see cpuid.h) located in avx512.cpp file.
As I remember some compilers (also clang LLVM ?) are controlled by external options for each file and 'fast' and 'base/old' AVX512 functions can not be mixed in single file.

For example 10-16bit integer H-resize can run on the old AVX512 CPUs but 8bit can not because 8bit permutex instruction not yet present in AVX512_BASE instructions sets. The direct software emulation of 8bit permute instruction works visibly slower and also not needed in such form because better to make separate 8bit function with 2x16bit outputs (required for later 16bit multiplication).

[jpsdr]

Finaly it may be faster than expected (doesn't mean it will be fast :D), so no remove.

[jpsdr]

Unable to build AVX512, i have an error on _bzhi_u64 unknow... :(

Edit:
WT...??!!! You can't build an x86 version of resample_avx512.hpp ???

Edit2:
Thanks to Chat GPT, i found how to handle this...

[jpsdr]

About AVX512 code, something bothering me...

As some core functions share the same code and are in the same file (for exemple resize_h_planar_uint8_avx512_permutex_vstripe_ks4_internal) for Fast and Base, you must compile the code with all the settings for Fast. Even if you have differents code pathes for intrinsic, as you've allowed the compiler to use all the options, how can you be sure it doesn't use them in the code ? Either in the "not intrinsic" part, or even in intrisic ? If you've allowed compiler to uses "better" CPU instructions than the one you choose in the intrinsic, what prevent it to use them, if it produces exactly what you've asked him to do, but better ? Even more, in the parts without intrinsic (standard C++ code) that are in the Base code path, how can you be sure it will not use Fast instructions, as it's allowed to do so ?

I hope i've explained properly my concern.

[DTL]

I am not sure how MSVC works with single /arch:AVX512 switch enabled. At least it can compile any directly used instructions (in the intrinsics). But I am not sure if it can use any supported by current used version of compiler AVX512 instructions to make better binary (and this may create not compatible with old AVX512 CPUs binary). It is subject to ask Microsoft (or robots or look in the MSDN).

With clang-LLVM users must manually enable each used AVX512 instruction set for each file and I think the compiler can not use any more AVX512 instruction sets (so binary will be compatible with CPUs up to defined AVX512 instruction sets). If some AVX512 instruction set is not enabled in compiler options and its intrinsic is used - the compiler will stop with error.

You may try to look in the cmake files how it prepares AVX512 switches for each _avx512X file for clang-LLVM (or other compilers like GCC ?).

The very widely used difference between AVX512_BASE (old) and AVX512_FAST (new) functions versions for convolution (resize and many other) is usage of old madd+add (sequence of 2 instructions)
result_lo = _mm512_add_epi32(result_lo, _mm512_madd_epi16(src_lo, coeff)); // a*b + c
for old AVX512 chips and usage of new AVX512_VNNI single instruction
result_0_31lo = _mm512_dpwssd_epi32(rounder, src_r0r1_0_31lo, coef_r0r1_0_31lo);
in the functions versions for AVX512_FAST.

So for AVX512_BASE binary compiler can not use VNNI single instruction optimization of FMA. How it is solved for MSVC builds I do not know. In program text I use templated function to select between old and VNNI methods and we expect the compiler will not optimize add+madd version of templated fucntion to VNNI instruction and make binary not compatible with non-VNNI CPUs.

Also this may be the reason to separate avx512b.cpp and avx512.hpp files for building with different (supported) compiler options. And for avx512b.cpp only AVX512_BASE AVX512 options enabled and it may be more fail safe from attempt to optimize for higher AVX512 instructions sets.

[jpsdr]

It seems i wasn't clear enough : it's the same code in the same file used for both Fast/Base, so it doesn't have different compiler options.
Also, i can only specify compiler options for .c/.cpp files, i can't for .h/.hpp. As they are included in files, the .h/.hpp is build with the options of the .c/.cpp calling it. To be able to build with LLVM, i had to activate ALL the avs options for both avx512 abd avx512b as they both include the .hpp. Finaly, i was able to build only activating ALL the options for all the AVX512 files.

[tormento]

Let me introduce the elephant in the room.

Isn’t time to add XYZ support to AVS+?

Powerful CPUs made DCP processing a thing even on “home” workstations.

What about that? [emoji4]

[PoeBear]

Is that not just due to the lack of external plug-ins? Ie if someone ported this VapourSynth plug-in (https://github.com/yuygfgg/Vapoursynth-MXFJ2KSource), is AVS+ missing some core functionality that it would still need?

[StainlessS]

Anything, anywhere, in a avisynth source that handles a colourspace (eg YV12), for resizing, colorspace conversion, blur, overlay, and any other colorspace handling [EDIT: pixel changes] you can think of,
would require a similar code path as eg YV12, we're talking a lot of new code here (I think).
XYZ addition is not a simple ask, lots of sweat and perhaps tears too.
[EDIT: Early Avisynth supported only YUY2 and RGB interleaved, it took some time to add even YV12]

Of course external plugins support would also require new code, excepting something like ClipClop which just juggles frame pointers and such,
does not process any colorspace data at all. [ClipClop only required additional code for debug/metrics text rendering for high bit depth, if you do not use text render, then always worked fine with HBD].
Most external plugins are not like ClipClop and would require added code path for XYZ stuff.
EDIT: Builtin functions like Trim() and Splice() would (like ClipClop) not (I think) need any changes.

EDIT:
I presume that XYZ would only exist in float format [not eg 8/10 bit], but dont know if might require 32 bit and/or 64 bit float [or additional bittages].

EDIT: Maybe all above is total bollocks ! :(

[Reel.Deel]

I updated to the latest test version (Avisynth+ v3.7.6pre-r4554 test build) and a simple mp_pipeline script that has always worked now returns an error.
MP_Pipeline: Unable to create slave process. Message:
(C:\Users\test\test.avs, line 12)

MP_Pipeline("""
### platform: win64
# this part of script will be run in a 64 bit process
Version(pixel_type="YV12")

# The line below is splitter
### ###

### platform: win32
HDRAGC(min_gain = 3.0, max_gain = 4.0, coef_gain=2)
### ###
""")

last

I downgraded both x64/x32 Avisynth to v3.7.5 and still the same problem. Then I downgraded the just x32 to v3.7.3 (version I has originally) and now the simple script works again. I'm using MP_Pipeline v0.23. HDRAGC (agc.dll) is in the plugins folder

[DTL]

" XYZ support to AVS+?"

For many filters without colour space conversion you can try to use it with current version and report if any issues exist. For better precision (if it is in linear transfer and HDR-like real range) - use float32 samples format. For SDR linear range the 12..14 bit expected to be enough.

[gispos]

I updated to the latest test version (Avisynth+ v3.7.6pre-r4554 test build) and a simple mp_pipeline script that has always worked now returns an error.


MP_Pipeline("""
### platform: win64
# this part of script will be run in a 64 bit process
Version(pixel_type="YV12")

# The line below is splitter
### ###

### platform: win32
HDRAGC(min_gain = 3.0, max_gain = 4.0, coef_gain=2)
### ###
""")

last

I downgraded both x64/x32 Avisynth to v3.7.5 and still the same problem. Then I downgraded the just x32 to v3.7.3 (version I has originally) and now the simple script works again. I'm using MP_Pipeline v0.23. HDRAGC (agc.dll) is in the plugins folder

MP_Pipeline works for me with this combination. I haven't tried anything newer. I didn't even know there was a version 0.23.

Avisynth 64bit, 3.7.5.0 r4523
Avisynth 32bit, 3.7.5.0 r4392
MP_Pipeline, 0.22
AGC.dll, 10.09.2006 18:37

Edit:
I just updated to MP_Pipeline v0.23 and it works.
In one of my recent Avisynth updates, the 32-bit Avisynth version contained 64-bit DLLs; perhaps that's a problem for you?

[Reel.Deel]

MP_Pipeline works for me with this combination. I haven't tried anything newer. I didn't even know there was a version 0.23.

Avisynth 64bit, 3.7.5.0 r4523
Avisynth 32bit, 3.7.5.0 r4392
MP_Pipeline, 0.22
AGC.dll, 10.09.2006 18:37

Edit:
I just updated to MP_Pipeline v0.23 and it works.
In one of my recent Avisynth updates, the 32-bit Avisynth version contained 64-bit DLLs; perhaps that's a problem for you?

Thanks for the feedback. I found the problem. I had an old yadifmod2 32bit plugin (v0.0.4-1 2016/07/04) in the autoload folder. Turns out in Avisynth > 3.7.3 it throws an error

Avisynth open failure:
'C:/Program Files (x86)/AviSynth+/plugins/yadifmod2.dll' plugin loading error:
Unknown exception


False flag, my bad, I'm rusty :p

[rgr]

Latest test build (https://github.com/pinterf/AviSynthPlus/releases/tag/v3.7.6pre-r4554)
Avisynth+ v3.7.6pre-r4554 test build (20260307)

[CODE]
20260307 3.7.5.r4554 (pre 3.7.6)
--------------------------------
- Fix: Working 'quality' parameter for 10-14 bits sources in YUV<->RGB conversions (was: blackness)


So far, I've been converting to RGB using Convertbits(32).ConvertToPlanarRGB (first converting to 32 bits to improve the accuracy of the conversion to RGB). Does ConvertToPlanarRGB(bits=32, quality=true) work the same way - first to 32 bits, then to RGB?
(I converted to YUV the same way, except I converted to 16 bits first.)

Edit: "Update https://avisynthplus.readthedocs.io/en/latest/avisynthdoc/corefilters/convert.html`
matrix syntax
ConvertToYUY2
bits and quality parameters"

The link on the page has an unnecessary apostrophe at the end.

"any->YUV conversions (See https://avisynthplus.readthedocs.io/en/latest/avisynthdoc/corefilters/convert.html`.
accept bits and quality parameters, similar to ConvertToPlanarRGB"

Here too.

[flossy_cake]

Small feature request for a frame prop please if possible.

In the Internal filter TemporalSoften (http://avisynth.nl/index.php/Soften#TemporalSoften), there is the parameter scenechange which sets the threshold for detecting scenechanges, which disables temporal softening for those frames to prevent blending over scenechanges.

Could we have an option to tell it to read from the _SceneChangePrev (http://avisynth.nl/index.php/Internal_functions#SceneChangePrev) property -- that way I can do my own better scenechange detection and copy it to that prop and TemporalSoften can use that instead.

I tried already to do it using ScriptClip but it doesn't work properly for technical reasons , things to do with desync and not being able to control the direction of blends either side of the scene which TemporalSoften manages internally when it detects a new scene

:thanks:

[real.finder]

The fftw library planner function is not thread safe. A simple instance of plugin can make it thread safe on its own. But when there are multiple instances and/or different plugins that use fftw DLLs, they can still mess into each others planner.

Avisynth's new v12 interface can help with this topic, if they use the same "global lock", different instances and plugins can safely work together.

https://avisynthplus.readthedocs.io/en/latest/avisynthdoc/FilterSDK/Cplusplus_api.html#cplusplus-acquiregloballock

The original issue:
https://github.com/AviSynth/AviSynthPlus/issues/444

now most of the important plugins get this fix, which are:-

http://avisynth.nl/index.php/FFT3DFilter

http://avisynth.nl/index.php/Dfttest

http://avisynth.nl/index.php/ManyPlus

https://github.com/HomeOfAviSynthPlusEvolution/neo_DFTTest

------------------------------

and here plugins that still dont get the fix:-

http://avisynth.nl/index.php/VariableBlur

http://avisynth.nl/index.php/Spectrogram

https://github.com/HomeOfAviSynthPlusEvolution/neo_FFT3D (at least not in the Releases)

https://github.com/Asd-g/AviSynth-FFTSpectrum

DePan, DePanEstimate and mvtools https://github.com/pinterf/mvtools/issues/65

http://avisynth.nl/index.php/KPassFilterCL


did I forget one? :)

[flossy_cake]

Small feature request for a frame prop please if possible.

In the Internal filter TemporalSoften (http://avisynth.nl/index.php/Soften#TemporalSoften), there is the parameter scenechange which sets the threshold for detecting scenechanges, which disables temporal softening for those frames to prevent blending over scenechanges.

Could we have an option to tell it to read from the _SceneChangePrev (http://avisynth.nl/index.php/Internal_functions#SceneChangePrev) property -- that way I can do my own better scenechange detection and copy it to that prop and TemporalSoften can use that instead.

I tried already to do it using ScriptClip but it doesn't work properly for technical reasons , things to do with desync and not being able to control the direction of blends either side of the scene which TemporalSoften manages internally when it detects a new scene

:thanks:


I realise no-one probably cares about this but I came up with a funny workaround which I thought was mildly amusing. Basically just insert a white flash frame with BlankClip(color=$ffffff) on the scenechanges and then set TemporalSoften's scene thresh to some thing really high like 0.65-0.80 so it only detects the white flash frames as scenechange. Then remove the white flash frames afterwards when AverageLuma() >= 235 and copy paste the next frame after it into the current one. It's crazy but it works. Just hope I don't get any desync with multithreading that could muck up the timing otherwise I could end up with random white flash frames in the video which would look a bit embarrassing :o

[pinterf]

New build! Thanks for your patience; I intentionally took a break from the forum for these weeks.
Avisynth r4565
https://github.com/pinterf/AviSynthPlus/releases/tag/v3.7.6pre-r4565

I was primarily focused on ColorBarsUHD (https://avisynthplus.readthedocs.io/en/latest/avisynthdoc/corefilters/colorbarsuhd.html). The post testing and validation work (avsresize, fmtconv, HDRTools) threw a few surprises and required some (a lot of) extra checks. I could easily spend another two weeks on this topic, but I’ve had enough for now.

For online documentation check https://avisynthplus.readthedocs.io/en/latest/

And for all the changes:
https://avisynthplus.readthedocs.io/en/latest/avisynthdoc/changelist376.html
20260328 3.7.5.r4565 (pre 3.7.6)
--------------------------------
- Add ColorbarsUHD
- Fix: LruCache self-tuning cache resize: raised ghost-hit threshold from ghosted>0 to ghosted>1,
preventing unbounded cache growth during backward seeking (Issue #379) and Bob/SeparateFields
access patterns (Issue #270).
- Fix: VideoFrame::Release() race condition in frame property cleanup under multi-threaded use
(stability fix for static-frame sources such as ColorBars, BlankClip).
- Histogram "color"/"color2": add targets100 parameter: the 6 ColorBars target boxes at 100% luma scale.
- Fix: YUV<->RGB conversion with quality=true at 10-14 bit sources (regression).
- Fix: Histogram "color"/"color2": "target" parameter no longer acts as master switch for "axes".

[gispos]

New build! Thanks for your patience; I intentionally took a break from the forum for these weeks.
Avisynth r4565
https://github.com/pinterf/AviSynthPlus/releases/tag/v3.7.6pre-r4565

20260328 3.7.5.r4565 (pre 3.7.6)
--------------------------------
- Fix: LruCache self-tuning cache resize: raised ghost-hit threshold from ghosted>0 to ghosted>1,
preventing unbounded cache growth during backward seeking (Issue #379) and Bob/SeparateFields
access patterns (Issue #270).


Now the strange behavior of QTGMC is also fixed: https://forum.doom9.org/showthread.php?t=186869
After playback a few frames back and the playback became significantly slower and started to jerk. Was worse with 32bit than with 64bit.

This fix is ​​very nice!

Edit:
Edit deleted. Error on my part.

[flossy_cake]

New build!
https://github.com/pinterf/AviSynthPlus/releases/tag/v3.7.6pre-r4565

- Fix: LruCache self-tuning cache resize: raised ghost-hit threshold from ghosted>0 to ghosted>1,
preventing unbounded cache growth during backward seeking (Issue #379) and Bob/SeparateFields
access patterns (Issue #270).


Hi pinterf and thankyou for all your hard work keeping Avisynth alive.

I am testing the x64-win7 build and these were my findings just now...

1. Text() crash with "\n"


ColorBars().KillAudio().ConvertToYV12()

return Text("\n", lsp=0) # crash
#return Text("\n", lsp=undefined()) # no crash
#return Text("n", lsp=0) # no crash


https://i.imgur.com/F9bfGAC.png


2. Issue #379 (https://github.com/AviSynth/AviSynthPlus/issues/379#issue-2056158737) appears unresolved


ColorBarsHD().KillAudio()
Spline36Resize(3840, 2160)
Spline36Resize(1920, 1080)
Spline36Resize(3840, 2160)
Spline36Resize(1920, 1080)
Spline36Resize(3840, 2160)
Spline36Resize(1920, 1080)
Spline36Resize(3840, 2160)
Spline36Resize(1920, 1080)
Prefetch(4)


Testing with AvsPMod:

1. Let it play for 200 frames, then pause
2. Hold left arrow to framestep backwards for 50 frames to frame 150
3. Press play

Result on my system:

https://i.imgur.com/SmbIvhR.png


Testing with MPC-HC + LAV (I think LAV Splitter is the one calling Avisynth.dll?)

1. Let it play for 200 frames, then pause
2. Tap "frame-step back" key multiple times (key can be set in MPC-HC's Options > Player > Keys)

Result on my system:

https://i.imgur.com/QD7ZANY.png


I will need to test this on my Win10 machine tomorrow with the standard x64 build.

That Win10 machine is more powerful and I think the smaller the CPU load, the less of an issue #379 becomes. That's why I had to put all that resizing of 4k->1080p->4k to put some load on the CPU otherwise there is no issue.

It's some weird thing when the CPU gets loaded up and you get some kind of cache thrashing?

Maybe we can try changing the ghost-hit threshold? Could you maybe add some testing options for me via SetCacheMode()? Like ghosted>2, ghosted>3, ghosted>4 etc ?

:thanks:

[flossy_cake]

Have tested on my win 10 machine (12600KF CPU).....Text("\n", lsp=0) does NOT crash on the Win10 build. Issue #379 still present.

[StvG]

Have tested on my win 10 machine (12600KF CPU).....Text("\n", lsp=0) does NOT crash on the Win10 build.

This crash does occur in both Windows 10 and Windows 11 when using AviSynth Intel compiler build.

[real.finder]

I think if someone wants to save memory then SetMemoryMax is not the right tool, because it can slow everything down.
It would probably be really useful to have an option in Avisynth to control the cache behavior, either more aggressively or not.
For me, please go in the aggressive direction. :)

maybe as 3 or CACHE_AGGRESSIVE http://avisynth.nl/index.php/Internal_functions#SetCacheMode ;)

and it will be nice if it get new auto default (maybe as -1 or CACHE_AUTO) option to see the available RAM and select the mode base on it, since mode 1 is use even less than the default (mode 0) (https://forum.doom9.org/showthread.php?p=1913370#post1913370)

[gispos]

maybe as 3 or CACHE_AGGRESSIVE http://avisynth.nl/index.php/Internal_functions#SetCacheMode ;)

and it will be nice if it get new auto default (maybe as -1 or CACHE_AUTO) option to see the available RAM and select the mode base on it, since mode 1 is use even less than the default (mode 0) (https://forum.doom9.org/showthread.php?p=1913370#post1913370)
Everything is going wrong for me today.
I just wanted to delete my previous posting. I loaded version 3.5 and thought it was the new one.

I tested CacheMode once with 1 and it got stuck, so I have to test again.
I will now delete my previous posting.

Edit:
I'm not happy with the cache behavior of the last version, in contrast to an old 3.5 version that caches 30 frames (bookmarks) without any problems, the new version already has problems with 10 frames. Frames that have already been cached are repeatedly thrown out of memory.

[real.finder]

now most of the important plugins get this fix, which are:-

http://avisynth.nl/index.php/FFT3DFilter

http://avisynth.nl/index.php/Dfttest

http://avisynth.nl/index.php/ManyPlus

https://github.com/HomeOfAviSynthPlusEvolution/neo_DFTTest

------------------------------

and here plugins that still dont get the fix:-

http://avisynth.nl/index.php/VariableBlur

http://avisynth.nl/index.php/Spectrogram

https://github.com/HomeOfAviSynthPlusEvolution/neo_FFT3D (at least not in the Releases)

https://github.com/Asd-g/AviSynth-FFTSpectrum

DePan, DePanEstimate and mvtools https://github.com/pinterf/mvtools/issues/65

http://avisynth.nl/index.php/KPassFilterCL


did I forget one? :)

Mvtools2 and DePans (with DePanEstimate) get the fix thanks to pinterf update!

[pinterf]

Thanks for the feedbacks, I'll slowly process them.
EDIT: zero-line-length Text crash is fixed (commit and test-release later)

[flossy_cake]

Is there something special about frame number 8 in relation to caching/multithreading?

Because inside GetFrame if I track nonlinear frame access by storing the current frame number into a class member variable and look at its value on the next frame, seeking often causes the prefetcher to make all these random GetFrames to frame number 8 even when I am thousands of frames ahead and have been playing video for 30 seconds.

Then after a while (sometimes up to 30 seconds) it settles down and stops randomly getting frame 8 and i get nice linear pattern again. But it's always frame 8, which is weird - is that hardcoded somehow? This is with MT_SERIALIZE and prefetch(4). This is also different to how it behaves inside a ScriptClip where I dont get these random requests of of frame number 8 (I still get some nonlinear access of course, that's part and parcel with multithreading, but it's not the same frame 8 again and again).

Grok sez...


Why frame 8 specifically?

The prefetcher in AviSynth+ does not pick random low numbers. It has internal logic that often causes it to (re)request a small fixed offset or a specific early frame during certain situations.
The number 8 is very common because:

Default prefetch window / step size
Many parts of the prefetch/cache system use small look-ahead values (often 4, 6, 8, or multiples of 8). Frame 8 is a frequent "probe" or "anchor" point when the prefetcher is trying to detect the access pattern at the start of a new range, after a cache miss, or when filling small gaps.

Cache validation / pattern detection
The core sometimes issues a request for frame N mod 8 or a small fixed frame like 8 when it wants to check consistency, validate the cache state, or when it temporarily "resets" its linear prediction after detecting a skip/jump. Frame 8 is a safe, small, non-negative number that is often used as a sentinel or test frame in the prefetch thread.

Alignment with internal constants
Several internal structures and default values in AviSynth+ (cache window sizes, prefetch distance, alignment-related code) are powers of 2 or multiples of 8. Frame 8 ends up being a "popular" low frame that the prefetcher falls back to when it needs to issue an out-of-order request for housekeeping reasons.

In short: Frame 8 is not special to your video — it's special to AviSynth+’s prefetch/cache implementation. It’s a recurring "noise" frame that many developers see when they add debug logging to MT_SERIALIZED filters.
Why it appears "sporadically"
It usually happens right after:

A cache miss higher up the chain
Small skips created by the prefetcher (e.g. requesting 96 → 98 → 100)
When the prefetch thread is trying to fill gaps or re-synchronize its prediction
After seeking or at the beginning of a new playback segment


edit: seem to have gleaned a pattern:


Prefetch(n) Frame number of random getframes

Prefetch(4) 8
Prefetch(2) 4
Prefetch(1) 2



So it seems to be double the prefetch value.

So let's say I'm using Prefetch(2) and I'm on frame 5000, the access pattern randomly goes like this:

5000, 5001, 5002, 5003, 4, 5004, 5005, 5006 etc...

And if I use Prefetch(4) it goes:

5000, 5001, 5002, 5003, 8, 5004, 5005, 5006 etc...

So I guess that's probably normal then?

________________________

edit: I guess it must be because the 3rd parameter of Prefetch is "frames" which defaults to threads*2. Coincidence?

edit: huh that's weird, if I get rid of a PropShow() which I'm using to track nonlinear access through frame properties, the problem goes away. It also goes away if I put a RequestLinear() between the MyFilter() and PropShow() (but not if I put it before MyFilter) suggesting PropShow is somehow the culprit who is triggering the frame cache to spaz out. Tried using Text() inside ScriptClip to print out my frame props and got the same issue, so it must be something to do with writing text overlays. Oh well at least I can dodge it now. And turning multithreading off altogether solves it too.

[wonkey_monkey]

Years later I'm still in the dark about the whole #define AVS_LINKAGE_DLLIMPORT / AVS_Linkage thing.

It seems you can do one of two things when compiling a plugin: either define AVS_LINKAGE_DLLIMPORT and link to avisynth.lib, or create your AVS_linkage variable which you set to the passed vectors parameter on initialisation.

Is one preferred over the other?

[Columbo]

Preferred by who? ;) Personally I prefer the second. Not for any reason but only because it's the method used by the sample plugins I modeled stuff on way back when.

[qyot27]

If you ask me, everyone should be strongly encouraged to move to opening the core with LoadLibrary or dlopen, and skip linking to it entirely. Examples of which are the way the FFMS2 C-plugin accesses the core or presumably by using this C++ wrapper that Asd-g published:
https://github.com/Asd-g/avisynthplus-c-api-dynamic-loader

[Reel.Deel]

did I forget one? :)

I think DeFreq should also be on the list. Someone recently added high bit-depth and additional colorspaces to it. Not sure if it has "the fix" - https://github.com/Kraz3yIvan/DeFreqPlus

[rgr]

https://github.com/HomeOfAviSynthPlusEvolution/neo_FFT3D (at least not in the Releases)


You can remove the neo_fft3d plugin and use:
1. QTGMC: denoiser="fft3dfilter"
2. TemporalDegrain2: limitFFT=11 (use fft3dfilter for limiting)

[real.finder]

I think DeFreq should also be on the list. Someone recently added high bit-depth and additional colorspaces to it. Not sure if it has "the fix" - https://github.com/Kraz3yIvan/DeFreqPlus

I dont think it has the fix, anyway, the new list

The fftw library planner function is not thread safe. A simple instance of plugin can make it thread safe on its own. But when there are multiple instances and/or different plugins that use fftw DLLs, they can still mess into each others planner.

Avisynth's new v12 interface can help with this topic, if they use the same "global lock", different instances and plugins can safely work together.

https://avisynthplus.readthedocs.io/en/latest/avisynthdoc/FilterSDK/Cplusplus_api.html#cplusplus-acquiregloballock

The original issue:
https://github.com/AviSynth/AviSynthPlus/issues/444

now most of the important plugins get this fix, which are:-

DePan, DePanEstimate and mvtools https://github.com/pinterf/mvtools/issues/65

http://avisynth.nl/index.php/FFT3DFilter

http://avisynth.nl/index.php/Dfttest

http://avisynth.nl/index.php/ManyPlus

https://github.com/HomeOfAviSynthPlusEvolution/neo_DFTTest

------------------------------

and here plugins that still dont get the fix:-

https://github.com/Kraz3yIvan/DeFreqPlus

http://avisynth.nl/index.php/VariableBlur

http://avisynth.nl/index.php/Spectrogram

https://github.com/HomeOfAviSynthPlusEvolution/neo_FFT3D (at least not in the Releases)

https://github.com/Asd-g/AviSynth-FFTSpectrum (edit: done in 1.1.2)

http://avisynth.nl/index.php/KPassFilterCL

[pinterf]

If you ask me, everyone should be strongly encouraged to move to opening the core with LoadLibrary or dlopen, and skip linking to it entirely. Examples of which are the way the FFMS2 C-plugin accesses the core or presumably by using this C++ wrapper that Asd-g published:
https://github.com/Asd-g/avisynthplus-c-api-dynamic-loader
Thanks to pointing to this repo, I have an in-queue C-plugin project (AvsInpaint) on which I can apply this and experiment with.

[pinterf]

While working on an update for AvsInPaint, and performing comparison tests with different formats I noticed inconsistent behavior between various Avisynth merge methods (Overlay, Layer, and Merge). I found pixel differences and issues with near-max pixel merge data.
Frustrated by these inconsistencies, I started a huge refactoring project to remove the redundant code base and rewrite this section. This was a much larger task than I thought originally, it triggered more and more places to look into, I found several unoptimized paths as well.
This has been my primary focus for the past month, but I am finally nearing completion. I'll be back then.

[tormento]

I'll be back then.
Thank you for your time and devotion.

[flossy_cake]

Is propGetAll (http://avisynth.nl/index.php/Internal_functions/YDifferenceToNext#propGetAll) working correctly for arrays, or is it only returning first element?


If the property value for a given key is an array again then "value" will be an array as well.


I can only access the first element:


BlankClip()
propSet("Array", ["one", "two", "three"])
PropShow()

ScriptClip(last,
\ """
props = propGetAll()
string = ""
for (i=0, ArraySize(props)-1){
string = string + String(props[i,0]) + " = " + String(props[i,1]) + "\n"
}

return Text(string, lsp=0, align=5)

\ """, after_frame=true, local=false)



https://i.imgur.com/dSnpJ5J.png


If props[0,1] was an array, then stringifying the array should just give an empty string, not "one".

I have tried props[0,1,1] or props[0][1][1] to try and retrieve "two" (second element of array) but it gives errors like array index out of bounds and such.

:thanks:

[gispos]

Is propGetAll (http://avisynth.nl/index.php/Internal_functions/YDifferenceToNext#propGetAll) working correctly for arrays, or is it only returning first element?

If props[0,1] was an array, then stringifying the array should just give an empty string, not "one".

I have tried props[0,1,1] or props[0][1][1] to try and retrieve "two" (second element of array) but it gives errors like array index out of bounds and such.

:thanks:

That's how it works for me, maybe there are other ways...
Oh, I see your question refers to propGetAll(). Then my answer is probably incorrect.
Using 'propGetAll' only returns the array of all Frame properties.
To iterate through the array of a single property, you need to use a separate 'for' loop on the array.

BlankClip()
propSet("Array", ["one", "two", "three"])
PropShow()

ScriptClip(last,
\ """
ar = propGetAsArray("Array")
str = ""
for (i=0, ArraySize(ar)-1){
str = str + ar[i] + "\n"
}

return Text(str, lsp=0, align=5)

\ """, after_frame=true, local=false)


https://i.postimg.cc/sD9J96dR/arrayget.jpg

[Boulder]

I updated my Avisynth installation since a long time, and noticed something weird with this combination and multithreading in AVS+. This is a 3840x2160 UHD source. When opened in VDub, the video looks like it was first downscaled heavily, then upscaled back. If I comment out the cropping or Prefetch, it works fine. Also if I use jpsdr's multithreaded resizers instead of internal ones, it works fine.

DGSource("W:\dovi\hot\hot.dgi")
Crop(0,44,-0,-44)
z_ConvertFormat(pixel_type="RGBPS", chromaloc_op="auto=>same", colorspace_op="auto:auto:auto:auto=>rgb:linear:same:full", cpu_type="AVX2", approximate_gamma=true, use_props=3)
BicubicResize(2560, 1384)
z_ConvertFormat(pixel_type="YUV420P16", cpu_type="AVX2", approximate_gamma=true, dither_type="none", use_props=4)
Prefetch(threads=12, frames=4)

I was able to determine that r4392 works but r4400 already has this issue.

[pinterf]

New build, Avisynth r4604
https://github.com/pinterf/AviSynthPlus/releases/tag/v3.7.6pre-r4604

Full change log:
https://avisynthplus.readthedocs.io/en/latest/avisynthdoc/changelist376.html

**20260518 3.7.5.r4604 (pre 3.7.6)**
--------------------------------
- Subtitle: add "placement" string parameter — chroma location hint for subsampled
YUV (4:2:0, 4:2:2, 4:1:1). When gdi=true, all three siting modes are supported:
"MPEG2"/"left" (default), "MPEG1"/"center", and "top_left" (UHD 4:2:0/4:2:2).
Default is read from the _ChromaLocation frame property, falling back to "left".
When gdi=false only "left" and "center" are implemented (same as Text).
- Subtitle: add "gdi" bool parameter. When false, Subtitle uses the built-in bitmap
font (Terminus) instead of Windows GDI rendering (same path as Text). Faster and
cross-platform compatible; placement is then limited to "left"/"center".
Default: true.
See at https://avisynthplus.readthedocs.io/en/latest/avisynthdoc/corefilters/subtitle.html
- Text: add "gdi" bool parameter (accepted, has no effect; API compatibility with
Subtitle — on non-Windows Subtitle is aliased to Text). Default: false.
- Subtitle Antialiaser (Windows GDI): GetAlphaRect() now has SSE4.1, AVX2, and
AVX512 SIMD implementations.
Internal mask buffer refactored to row-interleaved SoA layout for correct
chroma-placement-aware UV compositing via prepare_effective_mask_for_row
downsampling (8 MaskModes covering all subsampling ratios x siting variants;
SIMD rowprep dispatch matching Overlay/Layer).
- ShowCRC32: new "channels" (select planes), "mode" (0=combined, 1=per-plane), and
"showmode" (0=display, 1=display+frameprop, 2=frameprop only) parameters;
frame property "ShowCRC32" is an int64 array of unsigned 32-bit CRC values;
planar RGB(A) planes always in R,G,B,A order; packed formats with default parameters
hash the raw buffer unchanged (backward-compatible).
See at https://avisynthplus.readthedocs.io/en/latest/avisynthdoc/corefilters/showframes.html
- rstdoc: document Invert's chroma inversion behavior
See at https://avisynthplus.readthedocs.io/en/latest/avisynthdoc/corefilters/invert.html

20260506 3.7.5.r4589 (pre 3.7.6)
--------------------------------
TL;DR (since r4565):
- Layer/Overlay/Merge: weighted and masked-merge kernels unified into a shared family (C/SSE4.1/AVX2/NEON);
SIMD chroma-placement mask downsampling for 4:2:0/4:2:2/4:1:1 sources;
"placement" parameter for Overlay "blend" (new) and Layer (new "top_left" option added):
values are "mpeg2" (default), "mpeg1", "top_left".
See at https://avisynthplus.readthedocs.io/en/latest/avisynthdoc/corefilters/layer.html
- Layer: new "mulovr" mode (Overlay-compatible YUV multiply with luma-driven chroma desaturation);
new "top_left" placement option; legacy MMX/iSSE removed.
- Merge (non-masked) C/SIMD integer arithmetic unified.
- SIMD additions: AVX2/SSE4.1/NEON float chroma-mask row preparation; AVX2 float weighted merge.
- Further optimizations: Invert (AVX2/SSE2), ExtractX (direct packed-RGB path),
ConvertBits int depth-reduce C loop, PlanarRGB(A)->RGB32/64 (AVX2).
- SetFilterProp: conditional injection by filter argument value; capture mode;
SetFilterPropPassthrough helper; GetFilterProps() diagnostics function.
See also: https://github.com/AviSynth/AviSynthPlus/issues/393
and the online documentation at https://avisynthplus.readthedocs.io/en/latest/avisynthdoc/corefilters/setfilterprop.html

- Bug fixes: Text("\n") crash on zero-length lines; Layer no longer silently rounds odd x/y
offsets on subsampled YUV — accepts them as-is like Overlay; ceiling formula for
non-grid-aligned positions — last chroma column/row is no longer skipped in either filter.

[jpsdr]

Hi.
I've made some benchmarks (results will be posted later in appropriate thread), doing them i've noticed something...

I was benchmarking my MT resizer, when i've noticed something odd with the number of threads reported by AVSMeter.


Core i7 860 (4 cores / 8 threads)
-----------

Windows 7 x86
AVS 2.60 (2.6.0.6)

My MT resizer
threads=1: (1 thread)
threads=2: (3 threads)
threads=4: (5 threads)
threads=8: (9 threads)

This is old standard AVS, nothing odd.


Rizen Threadripper Pro 7975WX (32 cores / 64 threads)
-----------

Windows 10 x64
AVS+ 3.7.5 (r4657)

Internal Bicubic
No Prefetch: (68 threads)
Prefetch(4): (72 threads)
Prefetch(32): (100 threads)
Prefetch(64): (132 threads)

When i saw this insane number of threads with my MT resizer, i've tested with the internal resizer instead, to be sure it wasn't from my side.
The number of threads increase with the Prefetch value, expected, but why 68 threads in the first place when there is nothing with multithreadind enabled ?

For No Prefetch, the script is just :

ColorBars(3840,2160,"YV24").KillAudio().trim(0,199)
BicubicResize(width/2, height/2)

[pinterf]

Good question, maybe an autoloaded plugin is starting its thread pool?

[jpsdr]

Good remark.
Easy to check, so i deleted all the files in plugins64 and plugins64+ directories.
Same results, so all these threads are created by AVS+, even when no prefetch required.