AviSynth+ thread Vol.2

[Dogway]

Thanks both, yes this is not my area of expertise, but at least I could see the difference since ShowChannels() was called outside the runtime env.

I'm having another issue, with the goal to fetch past runtime variables I coded this (snippet from modded ScSelect_HBD() ):

ScriptClip( function [] () {

n = current_frame
propSet("_SceneChangePrevLast", n == 0 || SC != 0 ? 0 : current_frame, 0) # Last detected Start frame

SCP = n != 0 && SC != 0 ? PropGetInt(last,"_SceneChangePrevLast",offset=-1) : current_frame
propSet(last,"_SceneChangePrevLast",SCP, 0)
} )

This is kind of a readback, but I don't know if it's possible. Or I have to get out of the scriptclip, or even the function's scope. Basically I'm checking if current frame is not a SC, if so copy the previous frameproperty to current. Basically I want to hold the last SC frame number in "_SceneChangePrevLast" property until a new SC. I'm trying to avoid globals as generally they are not advised.


@pinterf: OMG, at last (https://github.com/AviSynth/AviSynthPlus/commit/e35f52e40a158c264e8a94bcc11f2025d006126d)?! Are you holding back for Xmas present :rolleyes: or is it still WIP? I predict a lot of filtering in 14-bits in the future if my 32Gb of RAM permit lol.

[pinterf]

Thanks both, yes this is not my area of expertise, but at least I could see the difference since ShowChannels() was called outside the runtime env.

I'm having another issue, with the goal to fetch past runtime variables I coded this (snippet from modded ScSelect_HBD() ):

ScriptClip( function [] () {

n = current_frame
propSet("_SceneChangePrevLast", n == 0 || SC != 0 ? 0 : current_frame, 0) # Last detected Start frame

SCP = n != 0 && SC != 0 ? PropGetInt(last,"_SceneChangePrevLast",offset=-1) : current_frame
propSet(last,"_SceneChangePrevLast",SCP, 0)
} )

This is kind of a readback, but I don't know if it's possible. Or I have to get out of the scriptclip, or even the function's scope. Basically I'm checking if current frame is not a SC, if so copy the previous frameproperty to current. Basically I want to hold the last SC frame number in "_SceneChangePrevLast" property until a new SC. I'm trying to avoid globals as generally they are not advised.


@pinterf: OMG, at last (https://github.com/AviSynth/AviSynthPlus/commit/e35f52e40a158c264e8a94bcc11f2025d006126d)?! Are you holding back for Xmas present :rolleyes: or is it still WIP? I predict a lot of filtering in 14-bits in the future if my 32Gb of RAM permit lol.
Yes, Luts are on the workbench. Some other cleanups are in progress.
Till then: presently I put an error if 1D or 2D Lut is not supported for a given bit depth. I suppose it's better if no fatal error is given but do the usual realtime Expr instead.

[Dogway]

I'm not a fan of error messages as they are disruptive, but in any case I already adapted ExTools to select proper lut int for given bitdepth. I'm very curious to see what kind of performance upgrades it brings, specially when there's pixel addressing involved in the code.

[DTL]

Do SetMaxCPU() support also AVX512 flags (disabling) ?

The file cpuid.h in the /avs includes to latest official MVtools https://github.com/pinterf/mvtools/blob/mvtools-pfmod/DePan/include/avs/cpuid.h lists

CPUF_AVX512F = 0x100000, // AVX-512 Foundation.
CPUF_AVX512DQ = 0x200000, // AVX-512 DQ (Double/Quad granular) Instructions
CPUF_AVX512PF = 0x400000, // AVX-512 Prefetch
CPUF_AVX512ER = 0x800000, // AVX-512 Exponential and Reciprocal
CPUF_AVX512CD = 0x1000000, // AVX-512 Conflict Detection
CPUF_AVX512BW = 0x2000000, // AVX-512 BW (Byte/Word granular) Instructions
CPUF_AVX512VL = 0x4000000, // AVX-512 VL (128/256 Vector Length) Extensions
CPUF_AVX512IFMA = 0x8000000, // AVX-512 IFMA integer 52 bit
CPUF_AVX512VBMI = 0x10000000,// AVX-512 VBMI


Wiki still only lists up to AVX2 - http://avisynth.nl/index.php/Internal_functions#SetMaxCPU

So exact question: if at CPU with AVX512F (and may be more) I set SetCPUMax(avx2) in the script - will it return IscriptEnviroment->GetCPUFlags() max AVX2 to the plugin ? And if not call this script function - will it return all AVX512 flags found ?

I do not have debugger at the system with AVX512F cpu and want to test plugin with AVX512 functions enabled and disabled (without creating special builds and adding more params).

[pinterf]

Do SetMaxCPU() support also AVX512 flags (disabling) ?

The file cpuid.h in the /avs includes to latest official MVtools https://github.com/pinterf/mvtools/blob/mvtools-pfmod/DePan/include/avs/cpuid.h lists

CPUF_AVX512F = 0x100000, // AVX-512 Foundation.
CPUF_AVX512DQ = 0x200000, // AVX-512 DQ (Double/Quad granular) Instructions
CPUF_AVX512PF = 0x400000, // AVX-512 Prefetch
CPUF_AVX512ER = 0x800000, // AVX-512 Exponential and Reciprocal
CPUF_AVX512CD = 0x1000000, // AVX-512 Conflict Detection
CPUF_AVX512BW = 0x2000000, // AVX-512 BW (Byte/Word granular) Instructions
CPUF_AVX512VL = 0x4000000, // AVX-512 VL (128/256 Vector Length) Extensions
CPUF_AVX512IFMA = 0x8000000, // AVX-512 IFMA integer 52 bit
CPUF_AVX512VBMI = 0x10000000,// AVX-512 VBMI


Wiki still only lists up to AVX2 - http://avisynth.nl/index.php/Internal_functions#SetMaxCPU

So exact question: if at CPU with AVX512F (and may be more) I set SetCPUMax(avx2) in the script - will it return IscriptEnviroment->GetCPUFlags() max AVX2 to the plugin ? And if not call this script function - will it return all AVX512 flags found ?

I do not have debugger at the system with AVX512F cpu and want to test plugin with AVX512 functions enabled and disabled (without creating special builds and adding more params).
No distinct values, but when you set AVX2 then all AVX512 flags are disabled.

[pinterf]

I'm not a fan of error messages as they are disruptive, but in any case I already adapted ExTools to select proper lut int for given bitdepth. I'm very curious to see what kind of performance upgrades it brings, specially when there's pixel addressing involved in the code.
I don't think pixel addressing is compatible with LUT theory.

[pinterf]

I do not have debugger at the system with AVX512F cpu and want to test plugin with AVX512 functions enabled and disabled (without creating special builds and adding more params).
btw, probably in a month I will have an i7 11th gen with AVX512 support. It is a beast. In this article https://www.anandtech.com/show/16535/intel-core-i7-11700k-review-blasting-off-with-rocket-lake/2 they actually had ~225-275W load which is rather huge for a 125W TDP unit.

[DTL]

btw, probably in a month I will have an i7 11th gen with AVX512 support.

It is very good for test plugins. At i5-11500 I today got strange error - can not found AVX2 for MVtools (using Avisynth+ 3.6.1 of the end of 2020). May be if AVX512 present it somehow clear AVX2 flag ? And no debugger at that system to check what is wrong. So currently simply disable that check.

"they actually had ~225-275W load"

For 2 kBytes register file and wide 512bit execution units in AVX512 mode it is really required more power. The register file of AVX2 is only 512 bytes so the AVX512 register file require about 4 times more power for switching. But it is the fastest memory on chip and very lovely for its performance.
Even at 10nm chip or thinner. So peaks power at AVX512 load was up to 290 W. I currently trying to make AVX512 version (mm512_dbsad) of lovely for many HD/UHD users block size 16x16 faster in compare with AVX2 (mm256_mpsadbw). At least already found - gathering load of ref and src is slower in compare with standard SIMD load. So need to redo AVX512 version from easy but slow gather load to standard SIMD load. 16x16 8bit block with ref and sad results fit freely in 2 kBytes AVX512 register file so I hope will be faster in compare with AVX2 version with reloading half of block from cache for process.

" when you set AVX2 then all AVX512 flags are disabled."

Addition: It looks SetCPUMax("avx2") works at i5-11500 as expected. At least AMD uProf can not disassemble AVX512 function and shows difference between enabled SetCPUMax("avx2") (use AVX2 version of function) and commented out SetCPUMax("avx2") string in the script. Looks like flags signalling about AVX512 working and the AVX512 version of function used. Unfortunately AMD uProf still can not load even symbols at the non-build system to display used functions names even with provided .pdb file and all equal paths to sources.

[pinterf]

A ColorBars.Info() will show you the detected CPU flags

[pinterf]

It's December.
Broken test 30 replaced with 32
With qyot27's interface additions
Avisynth+ 3.7.1 test build 32 (20211202) (https://drive.google.com/uc?export=download&id=1xv8zwctgJ16XThopha4TBsU-aw_FTUUP)
20211202 WIP
------------
- Fix: MinMax runtime filter family: check plane existance (e.g. error when requesting RPlaneMinMaxDifference on YV12)
- Fix: prevent x64 debug AviSynth builds from crashing in VirtualDub2 (opened through CAVIStreamSynth)
- ExtractY/U/V/R/G/B/A, PlaneToY: delete _ChromaLocation property. Set _ColorRange property to "full" if source is Alpha plane
- AviSynth interface additions: extend queryable internal environment properties.
Since Interface version 8 IScriptEnvironment::GetEnvProperty (Avisynth.h) and avs_get_env_property (avisynth_c.h)
interface functions can query some specific internal properties of AviSynth core. Thread count, etc..
These are mainly for internal use but some can be useful for plugins and external applications.
Each requested property has an identification number, they are found in avisynth.h and avisynth_c.h

This addition brought new properties to query: host system's endianness, interface version and bugfix subversion.
Relevant enum names start with AEP_ (cpp) or AVS_AEP_ (c) (AEP stands for Avisynth Environment Property)

Details:

AEP_HOST_SYSTEM_ENDIANNESS (c++) AVS_AEP_HOST_SYSTEM_ENDIANNESS (c)
Populated by 'little', 'big', or 'middle' based on what GCC and/or Clang report at compile time.

AEP_INTERFACE_VERSION (c++) AVS_AEP_INTERFACE_VERSION (c)
for requesting actual interface (main) version. An long awaited function.
So far the actual interface version could be queried only indirectly, with trial and error, by starting from e.g. 10 then
going back one by one until CheckVersion() did not report an exception/error code.

Even for V8 interface this was a bit tricky, the only way to detect was the infamous
has_at_least_v8 = true;
try { env->CheckVersion(8); } catch (const AvisynthError&) { has_at_least_v8 = false; }
method.

Now (starting from interface version 8.1) a direct version query is supported as well.
Of course this (one or two direct call only) is the future.
Programs or plugins which would like to identify older systems still must rely partially on the CheckVersion method.

CPP interface (through avisynth.h).

IScriptEnvironment *env = ...
int avisynth_if_ver = 6;
int avisynth_bugfix_ver = 0;
try {
avisynth_if_ver = env->GetEnvProperty(AEP_INTERFACE_VERSION);
avisynth_bugfix_ver = env->GetEnvProperty(AEP_INTERFACE_BUGFIX);
}
catch (const AvisynthError&) {
try { env->CheckVersion(8); avisynth_if_ver = 8; } catch (const AvisynthError&) { }
}
has_at_least_v8 = avisynth_if_ver >= 8; // frame properties, NewVideoFrameP, other V8 environment functions
has_at_least_v8_1 = avisynth_if_ver > 8 || (avisynth_if_ver == 8 && avisynth_bugfix_ver >= 1);
// 8.1: C interface frameprop access fixed, IsPropertyWritable/MakePropertyWritable support, extended GetEnvProperty queries
has_at_least_v9 = avisynth_if_ver >= 9; // future

C interface (through avisynth_c.h)

AVS_ScriptEnvironment *env = ...
int avisynth_if_ver = 6; // guessed minimum
int avisynth_bugfix_ver = 0;
int retval = avs_check_version(env, 8);
if (retval == 0) {
avisynth_if_ver = 8;
// V8 at least, we have avs_get_env_property but AVS_AEP_INTERFACE_VERSION query may not be supported
int retval = avs_get_env_property(env, AVS_AEP_INTERFACE_VERSION);
if(env->error == 0) {
avisynth_if_ver = retval;
retval = avs_get_env_property(env, AVS_AEP_INTERFACE_BUGFIX);
if(env->error == 0)
avisynth_bugfix_ver = retval;
}
}
has_at_least_v8 = avisynth_if_ver >= 8; // frame properties, NewVideoFrameP, other V8 environment functions
has_at_least_v8_1 = avisynth_if_ver > 8 || (avisynth_if_ver == 8 && avisynth_bugfix_ver >= 1);
// 8.1: C interface frameprop access fixed, IsPropertyWritable/MakePropertyWritable support, extended GetEnvProperty queries
has_at_least_v9 = avisynth_if_ver >= 9; // future


AEP_INTERFACE_BUGFIX (c++) AVS_AEP_INTERFACE_BUGFIX (c)
Denotes situations where there isn't a breaking change to the API,
but we need to identify when a particular change, fix or addition
to various API-adjacent bits might have occurred. Could also be
used when any new functions get added.

Since the number is modelled as 'changes since API bump' and
intended to be used in conjunction with checking the main
AVISYNTH_INTERFACE_VERSION, whenever the main INTERFACE_VERSION
gets raised, the value of INTERFACE_BUGFIX should be reset to zero.

The BUGFIX version is added here with already incremented once,
both because the addition of AVISYNTH_INTERFACE_BUGFIX_VERSION
itself would require it, but also because it's intended to signify
the fix to the C interface allowing frame properties to be read
back (which was the situation that spurred this define to exist
in the first place).

- CMake build environment:
While we do need the compiler to support C++17 features, we can
get by on older GCC using CMake 3.6 and -std=c++-1z with some other fixes.
CMAKE_CXX_STANDARD can be raised intelligently to 17 based on whether we detect CMake 3.8 or higher.
- Add AVISYNTHPLUS_INTERFACE_BUGFIX_VERSION

- Avisynth programming interface V8.1 or V9(?)):
Add 'MakePropertyWritable' to the IScriptEnvironment (CPP interface), avs_make_property_writable (C interface)
Add 'VideoFrame::IsPropertyWritable' (CPP interface), avs_is_property_writable (C interface)
(AviSynth interface version will be stepped to V9 in the release version?)

bool env->MakePropertyWritable(PVideoFrame *);
bool VideoFrame::IsPropertyWritable();

'MakePropertyWritable' is similar to 'MakeWritable' but it does not copy all bytes of the frame content in order to have a writable property set.

Reason: 'propSet' is a filter which does not alter frame content, but sets the given frame property in its each GetFrame.
So far it used MakeWritable to obtain a safely modifiable copy of frame properties, however - as a side-effect - full copy of frame content was performed.
(env->getFramePropsRW alone does not ensure a uniquely modifiable property set, it just obtains a pointer which can be used in the property setter functions)
(Note: frame properties of frames obtained by NewVideoFrame, MakeWritable and SubFrame are still safe to modify)
- Expr: when actual bit depth is too large for building LUT table, fallback to realtime mode.
lut_x 1D (realtime when 32 bit)
lut_xy 2D (realtime when 16 or 32 bits)
- Expr: allow 'f32' as internal autoscale target (was: i8, i10, i12, i14, i16 were accepted, only integers)
affects: 'scale_inputs' when "int", "intf", "all", "allf"
more on that (todo: refresh docs) http://avisynth.nl/index.php/Expr
- Expr: fix conversion factor (+correct chroma scaling) when integer-to-integer full-scale automatic range scaling was required
- New: Expr: new parameter integer 'lut'
integer 'lut' (default 0)
0: realtime expression
1: expression is converted to 1D lut (lut_x)
2: expression is converted to 2D lut (lut_xy)
Valid bit depths: lut=1 : 8-16 bits. lut=2 : 8-14 bits. Note: a 14 bit 2D lut needs (2^14)*(2^14)*2 bytes buffer in memory per plane (~1GByte)
In lut mode some keywords are forbidden in the expression: sx, sy, sxr, syr, frameno, time, relative pixel addressing

[VoodooFX]

Enhancement: xPlaneMin/Max/Median/MinMaxDifference runtime functions to accept old packed formats (RGB24/32/48/64 and YUY2)
Yesterday evening I was puzzled by the script's weirdness, but thanks to the Sandman I dreamed where bug is: unknowingly I was feeding YV12 to RPlaneMinMaxDifference, but there were no errors from it.

EDIT:
Actually I didn't checked it in the reality, didn't got my coffee yet, but I trust my dreams [it seems that there I can code better]. :D

[pinterf]

Yesterday evening I was puzzled by the script's weirdness, but thanks to the Sandman I dreamed where bug is: unknowingly I was feeding YV12 to RPlaneMinMaxDifference, but there were no errors from it.
:D
True, the sad truth is that this errorless behaviour is very true, but I fixed, and next time you'll get an error sized as big as an elephant :)

[VoodooFX]

I see that an elephant (https://github.com/AviSynth/AviSynthPlus/commit/13219ff9568bee29b4c1e7532db81f3ae0f40d88) is ready to be seen. :)
Is there any ETA on the official 3.7.1 release?

[pinterf]

I see that an elephant (https://github.com/AviSynth/AviSynthPlus/commit/13219ff9568bee29b4c1e7532db81f3ae0f40d88) is ready to be seen. :)
Is there any ETA on the official 3.7.1 release?
Not really. When it's done. Probably in month or two. Cleanups, documentation, tests. We'll discuss it at the project meeting in a pub. Eeeer, not. :)

[Dogway]

Thanks for the update!! Lots of core changes.

I was eager to test the lut calculations but I get slower speeds.

ConvertBits(14)
a=FlipHorizontal()
ex_blend(last,a,"overlay",0.5) # 246fps
Prefetch(6)

ConvertBits(16)
a=FlipHorizontal()
ex_blend(last,a,"overlay",0.5) # 310fps (270fps with Preftech(6) )
Prefetch(4)

Following I went to test ex_binarize(mode="otsu") which should see great improvements given I use a bunch of ex_lutxy() for the bins. Well, avspmod froze. I understand that some pre-scanning is to be made but I waited a few minutes for a 8-bit clip. Setting lut=0, or even deleting the lut args didn't make things better. Back to test29, no issues.

# 1080p@8-bits
ex_binarize(mode="otsu")

[DTL]

A ColorBars.Info() will show you the detected CPU flags

It is great idea. At i5-11500 it detects almost all possible AVX-512 extensions:
https://i3.imageban.ru/out/2021/12/01/ad931b334df8029caeb951ae475d3ecd.png

For 8/16bit samples most wanted is BW and VL for addition to F.

[pinterf]

Thanks for the update!! Lots of core changes.

I was eager to test the lut calculations but I get slower speeds.

ConvertBits(14)
a=FlipHorizontal()
ex_blend(last,a,"overlay",0.5) # 246fps
Prefetch(6)

ConvertBits(16)
a=FlipHorizontal()
ex_blend(last,a,"overlay",0.5) # 310fps (270fps with Preftech(6) )
Prefetch(4)

Following I went to test ex_binarize(mode="otsu") which should see great improvements given I use a bunch of ex_lutxy() for the bins. Well, avspmod froze. I understand that some pre-scanning is to be made but I waited a few minutes for a 8-bit clip. Setting lut=0, or even deleting the lut args didn't make things better. Back to test29, no issues.

# 1080p@8-bits
ex_binarize(mode="otsu")
O.k. this is why it is called test.
I noticed too that Lut is not necessarily quicker, probably because of the extra memory round? Probably with larger e.g. 4K clip sizes?

EDIT: thanks for the report.
Though I wrote the world's quickest infinite loop, it didn't manage to return from a function :) New build is being produced.

[Dogway]

Ah no problem, take your time, this is a long due feature, as long as it works like in masktools2 I'm fine.

[pinterf]

Test30 replacement
Test31 removed until it really works. Thanks for Dogway for the feedback

[pinterf]

Thanks for the update!! Lots of core changes.

I was eager to test the lut calculations but I get slower speeds.

ConvertBits(14)
a=FlipHorizontal()
ex_blend(last,a,"overlay",0.5) # 246fps
Prefetch(6)

ConvertBits(16)
a=FlipHorizontal()
ex_blend(last,a,"overlay",0.5) # 310fps (270fps with Preftech(6) )
Prefetch(4)


Are you sure ex_blend is using lutx or lutxy?

[Dogway]

Yes, lutxy, the lower block is for reference. Anyway I haven't updated GradePack for some months, so better test with ex_binarize(mode="otsu") for now. Will check test31.

Testing with test31:
expr("x 80 > 255 0 ?","",lut=1)
Ok with lut=0

[pinterf]

Next round with qyot27's interface additions
Avisynth+ 3.7.1 test build 32 (20211202) (https://drive.google.com/uc?export=download&id=1xv8zwctgJ16XThopha4TBsU-aw_FTUUP)
20211202 WIP
------------
- Fix: MinMax runtime filter family: check plane existance (e.g. error when requesting RPlaneMinMaxDifference on YV12)
- Fix: prevent x64 debug AviSynth builds from crashing in VirtualDub2 (opened through CAVIStreamSynth)
- ExtractY/U/V/R/G/B/A, PlaneToY: delete _ChromaLocation property. Set _ColorRange property to "full" if source is Alpha plane
- AviSynth interface additions: extend queryable internal environment properties.
Since Interface version 8 IScriptEnvironment::GetEnvProperty (Avisynth.h) and avs_get_env_property (avisynth_c.h)
interface functions can query some specific internal properties of AviSynth core. Thread count, etc..
These are mainly for internal use but some can be useful for plugins and external applications.
Each requested property has an identification number, they are found in avisynth.h and avisynth_c.h

This addition brought new properties to query: host system's endianness, interface version and bugfix subversion.
Relevant enum names start with AEP_ (cpp) or AVS_AEP_ (c) (AEP stands for Avisynth Environment Property)

Details:

AEP_HOST_SYSTEM_ENDIANNESS (c++) AVS_AEP_HOST_SYSTEM_ENDIANNESS (c)
Populated by 'little', 'big', or 'middle' based on what GCC and/or Clang report at compile time.

AEP_INTERFACE_VERSION (c++) AVS_AEP_INTERFACE_VERSION (c)
for requesting actual interface (main) version. An long awaited function.
So far the actual interface version could be queried only indirectly, with trial and error, by starting from e.g. 10 then
going back one by one until CheckVersion() did not report an exception/error code.

Even for V8 interface this was a bit tricky, the only way to detect was the infamous
has_at_least_v8 = true;
try { env->CheckVersion(8); } catch (const AvisynthError&) { has_at_least_v8 = false; }
method.

Now (starting from interface version 8.1) a direct version query is supported as well.
Of course this (one or two direct call only) is the future.
Programs or plugins which would like to identify older systems still must rely partially on the CheckVersion method.

CPP interface (through avisynth.h).

IScriptEnvironment *env = ...
int avisynth_if_ver = 6;
int avisynth_bugfix_ver = 0;
try {
avisynth_if_ver = env->GetEnvProperty(AEP_INTERFACE_VERSION);
avisynth_bugfix_ver = env->GetEnvProperty(AEP_INTERFACE_BUGFIX);
}
catch (const AvisynthError&) {
try { env->CheckVersion(8); avisynth_if_ver = 8; } catch (const AvisynthError&) { }
}
has_at_least_v8 = avisynth_if_ver >= 8; // frame properties, NewVideoFrameP, other V8 environment functions
has_at_least_v8_1 = avisynth_if_ver > 8 || (avisynth_if_ver == 8 && avisynth_bugfix_ver >= 1);
// 8.1: C interface frameprop access fixed, IsPropertyWritable/MakePropertyWritable support, extended GetEnvProperty queries
has_at_least_v9 = avisynth_if_ver >= 9; // future

C interface (through avisynth_c.h)

AVS_ScriptEnvironment *env = ...
int avisynth_if_ver = 6; // guessed minimum
int avisynth_bugfix_ver = 0;
int retval = avs_check_version(env, 8);
if (retval == 0) {
avisynth_if_ver = 8;
// V8 at least, we have avs_get_env_property but AVS_AEP_INTERFACE_VERSION query may not be supported
int retval = avs_get_env_property(env, AVS_AEP_INTERFACE_VERSION);
if(env->error == 0) {
avisynth_if_ver = retval;
retval = avs_get_env_property(env, AVS_AEP_INTERFACE_BUGFIX);
if(env->error == 0)
avisynth_bugfix_ver = retval;
}
}
has_at_least_v8 = avisynth_if_ver >= 8; // frame properties, NewVideoFrameP, other V8 environment functions
has_at_least_v8_1 = avisynth_if_ver > 8 || (avisynth_if_ver == 8 && avisynth_bugfix_ver >= 1);
// 8.1: C interface frameprop access fixed, IsPropertyWritable/MakePropertyWritable support, extended GetEnvProperty queries
has_at_least_v9 = avisynth_if_ver >= 9; // future


AEP_INTERFACE_BUGFIX (c++) AVS_AEP_INTERFACE_BUGFIX (c)
Denotes situations where there isn't a breaking change to the API,
but we need to identify when a particular change, fix or addition
to various API-adjacent bits might have occurred. Could also be
used when any new functions get added.

Since the number is modelled as 'changes since API bump' and
intended to be used in conjunction with checking the main
AVISYNTH_INTERFACE_VERSION, whenever the main INTERFACE_VERSION
gets raised, the value of INTERFACE_BUGFIX should be reset to zero.

The BUGFIX version is added here with already incremented once,
both because the addition of AVISYNTH_INTERFACE_BUGFIX_VERSION
itself would require it, but also because it's intended to signify
the fix to the C interface allowing frame properties to be read
back (which was the situation that spurred this define to exist
in the first place).

- CMake build environment:
While we do need the compiler to support C++17 features, we can
get by on older GCC using CMake 3.6 and -std=c++-1z with some other fixes.
CMAKE_CXX_STANDARD can be raised intelligently to 17 based on whether we detect CMake 3.8 or higher.
- Add AVISYNTHPLUS_INTERFACE_BUGFIX_VERSION

- Avisynth programming interface V8.1 or V9(?)):
Add 'MakePropertyWritable' to the IScriptEnvironment (CPP interface), avs_make_property_writable (C interface)
Add 'VideoFrame::IsPropertyWritable' (CPP interface), avs_is_property_writable (C interface)
(AviSynth interface version will be stepped to V9 in the release version?)

bool env->MakePropertyWritable(PVideoFrame *);
bool VideoFrame::IsPropertyWritable();

'MakePropertyWritable' is similar to 'MakeWritable' but it does not copy all bytes of the frame content in order to have a writable property set.

Reason: 'propSet' is a filter which does not alter frame content, but sets the given frame property in its each GetFrame.
So far it used MakeWritable to obtain a safely modifiable copy of frame properties, however - as a side-effect - full copy of frame content was performed.
(env->getFramePropsRW alone does not ensure a uniquely modifiable property set, it just obtains a pointer which can be used in the property setter functions)
(Note: frame properties of frames obtained by NewVideoFrame, MakeWritable and SubFrame are still safe to modify)
- Expr: when actual bit depth is too large for building LUT table, fallback to realtime mode.
lut_x 1D (realtime when 32 bit)
lut_xy 2D (realtime when 16 or 32 bits)
- Expr: allow 'f32' as internal autoscale target (was: i8, i10, i12, i14, i16 were accepted, only integers)
affects: 'scale_inputs' when "int", "intf", "all", "allf"
more on that (todo: refresh docs) http://avisynth.nl/index.php/Expr
- Expr: fix conversion factor (+correct chroma scaling) when integer-to-integer full-scale automatic range scaling was required
- New: Expr: new parameter integer 'lut'
integer 'lut' (default 0)
0: realtime expression
1: expression is converted to 1D lut (lut_x)
2: expression is converted to 2D lut (lut_xy)
Valid bit depths: lut=1 : 8-16 bits. lut=2 : 8-14 bits. Note: a 14 bit 2D lut needs (2^14)*(2^14)*2 bytes buffer in memory per plane (~1GByte)
In lut mode some keywords are forbidden in the expression: sx, sy, sxr, syr, frameno, time, relative pixel addressing

[Dogway]

:thanks:

Everything seems to be working fine now. A bit sad because I lost one month of notes and drafts (the AvsPmod session file got cleansed : (

In big scripts like ex_binarize(mode="otsu") I don't see speed improvements but probably this is expected.

While updating the functions I got into a situation where the yexpr needs "intf" scale_inputs, and uexpr/vexpr requires "none" (or for 32-bits "none" and "floatUV" respectively). I could work around this by for example adding i16 (for 16-bit inputs) to the chroma expr, but this was slow so I baked the expression when 32-bit is being fed in.

# normalize to full range and back for limited range sources
# works fine over int HBD types and 32-bit float.
rangePCc = tv ? "f32 x 255 240 / *" : "f32 x"
rangeTVc = tv ? "240 255 / * " : " "

Another issue I found is ExtractU() sometimes doesn't work for float inputs.

[gispos]

A bit sad because I lost one month of notes and drafts (the AvsPmod session file got cleansed : (

Due to an error in the program? or how did that happen?
If the files are so important (one month of work) then the best thing to do is to make backup copies of them.

[Dogway]

I think it was my fault since I didn't enable avspmod's backups, but I'm not sure it works as I think it does.
I had a session from 24th of October, that's something.
I simply was writing something somewhere while avspmod launched, some letter got typed into the script editor and the last session got reset.
Is there an option for the current session to be stored as a "ghost" file (like Office does) until avspmod is closed?

[gispos]

I think it was my fault since I didn't enable avspmod's backups, but I'm not sure it works as I think it does.
I had a session from 24th of October, that's something.
I simply was writing something somewhere while avspmod launched, some letter got typed into the script editor and the last session got reset.
Is there an option for the current session to be stored as a "ghost" file (like Office does) until avspmod is closed?
The _last_session_.ses is overwritten with the current session.
If the backup is active, It will be overwritten every time the script has changed and the clip is re-initialized.
Hence the name 'last_session' :)

So if a session is important to you, you have to save this session yourself 'manually' with your own name.
A session you have saved yourself is never changed.
A shadow copy is therefore unnecessary, but I will think about whether it makes sense to save the currently opened '_last_session_.ses'.

Sorry Ferenc, I've finished. :)

[Dogway]

Manual save manual load, too manual for a program.

@pinterf: I noticed pixel addressing isn't being scaled with 'scale_inputs', at least for f32. Is it possible to include scaling the clip before pixels are being fetched?
Concerning 'lut' I didn't test but maybe it would come handy if it works when expression is scaled to supported bitdepth.
Also noticed there isn't a bool frame property, not sure if there's a reason for this.

[pinterf]

scaling for pixels obtained from rel.addressing: good catch, done in my work copy.
bool frame properties do not exist (they did not exist in VapourSynth), one must use integers for that purpose.

[pinterf]

@gispos: when you'd like to recognize Avisynth versions which have a fixed C interface for getting frame properties, you can do that since latest 3.7.1 test32.
See changelog for an example how to identify plain v8, then 8.1 or future 9 interface versions.

[StainlessS]

one must use integers for that purpose.
One advantage that tactic might have over bool is,
if originally some property defined as bool and you decide it aint enough and you want change to int ...
no longer a prob.

[Dogway]

It can be applied to _Interlaced, _ColorRange (there's also "extended" but oh well), _SceneChange, etc. Numbers imply several options and also forces you check against a number if either true or false.

[StainlessS]

That is true, [life's a bitch - and then you die]

[pinterf]

Avisynth+ 3.7.1 test build 33 (20211205) (https://drive.google.com/uc?export=download&id=1VxaMZaCee9kddzGOJ_0DTTJLgd8b4pSr)
I think it's time to make a feature freeze.
20211205 WIP
------------
- New array modifier function: ArraySet

For memo here is the list of avaliable array manipulator functions
- ArrayAdd - append
- ArrayDel - delete at position
- ArrayIns - insert before position
- ArraySet - replace at position

ArrayIns
^^^^^^^^

ArrayIns(array_to_mod, value_to_insert, index1 [, index2, index3...])

Insert a value into an array or into its subarray.
Returns a new array with value_to_insert inserted into array_to_mod (1D array) or array_to_mod[index1 (, index2, index3...)] (multi-dimensional array)
The indexes point to the insertion point. Index 0 will insert at the beginning of the array.
Index (ArraySize) will insert after the last element (same as ArrayAdd - append)
Original array (as with the other functions) remains untouched.

ArrayAdd
^^^^^^^^

ArrayAdd(array_to_mod, value_to_append [, index1, index2, index3...])

Appends value to the end of an array or its subarray
Returns a new array with value_to_append appended to array_to_mod (1D array) or array_to_mod[index1 (, index2, index3...)] (multi-dimensional array).
Original array (as with the other functions) remains untouched.

ArrayDel
^^^^^^^^

ArrayDel(array_to_mod, index1 (, index2, index3...])

Returns a new array in which the requested position was deleted.
Original array (as with the other functions) remains untouched.

ArraySet
^^^^^^^^

ArraySet(array_to_mod, replacement_value, index1 [, index2, index3...])

Returns a new array with array_to_mod[index1 (, index2, index3...)] = replacement_value
Original array (as with the other functions) remains untouched.

- Array modifier functions to allow multidimensional subarray indexes

Example:

ColorbarsHD()
# array indexes are zero based
a = []
a=ArrayAdd(a,[1,2]) # [[1,2]]
a=ArrayIns(a,3,0) # [3,[1,2]]
a=ArrayAdd(a,"s1") # [3,[1,2],"s1"]
a=ArrayAdd(a,"s2") # [3,[1,2],"s1","s2"]
a=ArrayDel(a,2) # [3,[1,2],"s2"]
a=ArraySet(a,"g",1,0) # [3,["g",2],"s2"]
a=ArrayAdd(a,"h",1) # [3,["g",2,"h"],"s2"]
a=ArrayAdd(a,[10,11,12],1) # append to (1) -> [3,["g",2,"h",[10,11,12]],"s2"]
a=ArrayDel(a,1,3,0) # del from (1,3,0) -> [3,["g",2,"h",[11,12]],"s2"]
a=ArrayAdd(a,"added") # [3,["g",2,"h",[11,12]],"s2","added"]
a=ArrayAdd(a,["yet","another","sub"]) # [3,["g",2,"h",[11,12]],"s2","added",["yet","another","sub"]]
x=a[0] #3
x=a[1,0] #g
x=a[1,2] #h
x=a[1,3,1] #12
x=a[3] #"added"
x=a[4,1] #"another"
SubTitle("x = " + String(x) + " Size=" + String(a.ArraySize()))

- Expr: allow auto scaling effect on pixels obtained from relative addressing
- ConvertBits: ordered dither: possible to dither down with more than 8 bits difference like in
Clip16.ConvertBits(8, dither=0, dither_bits=4)
Such conversion is made in two phases. First the clip is converted to (dither_bits+8) bits; in the above example it is 12.
If the temporary bit depth would be odd (no 9 or 11 bit support in Avisynth+) then it is made even.
bit depth that differs in only 8 bits for the target. Then this intermediate clip is converted to the required end target.
- Quicker ClearProperties and CopyProperties filters (by using MakePropertyWritable instead of MakeWritable).

[FranceBB]

Thanks for the new version!


I think it's time to make a feature freeze.

Oh, so this is gonna be the extensive testing part and Test 33 is the release candidate before the stable release?! :D
And perhaps the stable will be released for Christmas?

https://c.tenor.com/6gyNP9vCJH4AAAAC/its-a-christmas-miracle-christmas-miracle.gif

[gispos]

Manual save manual load, too manual for a program.

:rolleyes: It is automatically backed up and restored, and if you want to back up something, no program can only do it with good talk.
But I can also save all sessions with the date... and nobody looks through it afterwards and knows what is in the files.

There will be a backup copy of the last session and a previous one.

@gispos: when you'd like to recognize Avisynth versions which have a fixed C interface for getting frame properties, you can do that since latest 3.7.1 test32.
See changelog for an example how to identify plain v8, then 8.1 or future 9 interface versions.
Thanks Ferenc, had already read that.
And since this is not present in older versions, I had already determined it this way:
From header version 8 only the matrix is read (if necessary with Eval), and from version 3.71 the properties are also read.

Edit: Where can I find your integer properties value conversion to string?

[Dogway]

pinterf, do you know if Clip32.Expr(a,b,"x y - abs 128 > 255 0 ?", lut=2, scale_inputs="all") is a compatible lut expression? I think this is not documented.

[pinterf]

pinterf, do you know if Clip32.Expr(a,b,"x y - abs 128 > 255 0 ?", lut=2, scale_inputs="all") is a compatible lut expression? I think this is not documented.
Valid, because when the given bit depth is not available for LUT mode it fallbacks to realtime (lut=0). You said you didn't like error messages

- Expr: when actual bit depth is too large for building LUT table, fallback to realtime mode.
lut_x 1D (realtime when 32 bit)
lut_xy 2D (realtime when 16 or 32 bits)

[Dogway]

Sure, it's not about error messages but if it is possible to use lut calculations on scaled_inputs. The above for example could use an 8-bit lut table.

[StainlessS]

Doggie, Probably slower than realtime. [I think]

[Dogway]

test33 crashes avspmod with a simple bilinearresize(), same in avsmeter, has this to do with the interface change in test32? Strange because I was using test32 without issues.

[qyot27]

[Some? Most?] of the filter sources that have x86 intrinsics in them are in the process of being de-duplicated. It would seem there's a regression introduced in there somewhere, since a build with the intrinsics disabled works as expected:
$ mpv test.avs
mpv: ../avs_core/filters/resample.cpp:424: FilteredResizeH::FilteredResizeH(PClip, double, double, int, ResamplingFunction*, IScriptEnvironment*): Assertion `0' failed.
Aborted (core dumped)
(gdb) bt
#0 __pthread_kill_implementation (no_tid=0, signo=6, threadid=140737101604416)
at pthread_kill.c:44
#1 __pthread_kill_internal (signo=6, threadid=140737101604416)
at pthread_kill.c:80
#2 __GI___pthread_kill (threadid=140737101604416, signo=signo@entry=6)
at pthread_kill.c:91
#3 0x00007ffff5e5f476 in __GI_raise (sig=sig@entry=6)
at ../sysdeps/posix/raise.c:26
#4 0x00007ffff5e457b7 in __GI_abort () at abort.c:79
#5 0x00007ffff5e456db in __assert_fail_base
(fmt=0x7ffff5ff9770 "%s%s%s:%u: %s%sAssertion `%s' failed.\n%n", assertion=0x7fffd29785a1 "0", file=0x7fffd2978580 "../avs_core/filters/resample.cpp", line=424, function=<optimized out>) at assert.c:92
#6 0x00007ffff5e56e26 in __GI___assert_fail
(assertion=0x7fffd29785a1 "0", file=0x7fffd2978580 "../avs_core/filters/resample.cpp", line=424, function=0x7fffd2978518 "FilteredResizeH::FilteredResizeH(PClip, double, double, int, ResamplingFunction*, IScriptEnvironment*)")
at assert.c:101
#7 0x00007fffd25366c3 in FilteredResizeH::FilteredResizeH(PClip, double, double, int, ResamplingFunction*, IScriptEnvironment*) ()
at /usr/local/lib/libavisynth.so
#8 0x00007fffd25397e7 in FilteredResize::CreateResizeH(PClip, double, double, int, ResamplingFunction*, IScriptEnvironment*) ()
--Type <RET> for more, q to quit, c to continue without paging--
at /usr/local/lib/libavisynth.so
#9 0x00007fffd2539d9f in FilteredResize::CreateResize(PClip, int, int, AVSValue const*, ResamplingFunction*, IScriptEnvironment*) ()
at /usr/local/lib/libavisynth.so
#10 0x00007fffd253a182 in FilteredResize::Create_BilinearResize(AVSValue, void*, IScriptEnvironment*) () at /usr/local/lib/libavisynth.so
#11 0x00007fffd223c19b in FilterConstructor::InstantiateFilter() const ()
at /usr/local/lib/libavisynth.so
#12 0x00007fffd2276b73 in ScriptEnvironment::Invoke_(AVSValue*, AVSValue const&, char const*, Function const*, AVSValue const&, char const* const*, InternalEnvironment*, bool) () at /usr/local/lib/libavisynth.so
#13 0x00007fffd22822a2 in ThreadScriptEnvironment::Invoke_(AVSValue*, AVSValue const&, char const*, Function const*, AVSValue const&, char const* const*) ()
at /usr/local/lib/libavisynth.so
#14 0x00007fffd22d88a0 in ExpFunctionCall::Evaluate(IScriptEnvironment*) ()
at /usr/local/lib/libavisynth.so
#15 0x00007fffd22d5037 in ExpExceptionTranslator::Evaluate(IScriptEnvironment*)
() at /usr/local/lib/libavisynth.so
#16 0x00007fffd22d53cf in ExpLine::Evaluate(IScriptEnvironment*) ()
at /usr/local/lib/libavisynth.so
#17 0x00007fffd22d4f8e in ExpSequence::Evaluate(IScriptEnvironment*) ()
at /usr/local/lib/libavisynth.so
#18 0x00007fffd22d4e25 in ExpRootBlock::Evaluate(IScriptEnvironment*) ()
--Type <RET> for more, q to quit, c to continue without paging--
at /usr/local/lib/libavisynth.so
#19 0x00007fffd22dbe15 in Eval(AVSValue, void*, IScriptEnvironment*) ()
at /usr/local/lib/libavisynth.so
#20 0x00007fffd223c19b in FilterConstructor::InstantiateFilter() const ()
at /usr/local/lib/libavisynth.so
#21 0x00007fffd2276ae5 in ScriptEnvironment::Invoke_(AVSValue*, AVSValue const&, char const*, Function const*, AVSValue const&, char const* const*, InternalEnvironment*, bool) () at /usr/local/lib/libavisynth.so
#22 0x00007fffd2281c03 in ThreadScriptEnvironment::Invoke(char const*, AVSValue, char const* const*) () at /usr/local/lib/libavisynth.so
#23 0x00007fffd22dcec9 in Import(AVSValue, void*, IScriptEnvironment*) ()
at /usr/local/lib/libavisynth.so
#24 0x00007fffd223c19b in FilterConstructor::InstantiateFilter() const ()
at /usr/local/lib/libavisynth.so
#25 0x00007fffd2276ae5 in ScriptEnvironment::Invoke_(AVSValue*, AVSValue const&, char const*, Function const*, AVSValue const&, char const* const*, InternalEnvironment*, bool) () at /usr/local/lib/libavisynth.so
#26 0x00007fffd2281c03 in ThreadScriptEnvironment::Invoke(char const*, AVSValue, char const* const*) () at /usr/local/lib/libavisynth.so
#27 0x00007fffd22aa949 in avs_invoke () at /usr/local/lib/libavisynth.so
#28 0x000055555574e4ba in ()
#29 0x0000555555c80201 in ()
#30 0x0000555555815984 in ()
--Type <RET> for more, q to quit, c to continue without paging--
#31 0x000055555580dc0d in ()
#32 0x000055555580e49a in ()
#33 0x000055555586f716 in ()
#34 0x00007ffff5eb1927 in start_thread (arg=<optimized out>)
at pthread_create.c:435
#35 0x00007ffff5f419e4 in clone ()
at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100

[tormento]

I am having crashes too with some scripts and not others.

The following

SetMemoryMax()
SetFilterMTMode("DEFAULT_MT_MODE", 2)
LoadPlugin("D:\Eseguibili\Media\DGDecNV\DGDecodeNV.dll")
DGSource("F:\In\1_58 Ieri oggi domani\ieri.dgi",ct=132,cb=132,cl=0,cr=0)
CompTest24(1)
ConvertBits(16)
SMDegrain (tr=3, thSAD=300, refinemotion=true, contrasharp=false, PreFilter=6, plane=4, chroma=true)
fmtc_bitdepth (bits=8,dmode=8)
Prefetch(6)


gives me

System exception - Access Violation
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.3.9d~Dogway.avsi, line 901)
(D:/Programmi/Media/AviSynth+/plugins64/SMDegrain-3.3.9d~Dogway.avsi, line 228)

[FranceBB]

test33 crashes avspmod with a simple bilinearresize(), same in avsmeter, has this to do with the interface change in test32? Strange because I was using test32 without issues.

Yep, I've just tried with a simple:

ColorBars(848, 480, pixel_type="YV12")

BilinearResize(1920, 1080)

crashes immediately.

Even worse, getting rid of BilinearResize() makes it crash just as well, for instance:

ColorBars(848, 480, pixel_type="YV12")

makes AVSPmod x64 crash.

Trying with:


SetMaxCPU("none")

ColorBars(848, 480, pixel_type="YV12")


works

https://i.imgur.com/wI6IW7z.png

same goes for:


SetMaxCPU("none")

ColorBars(848, 480, pixel_type="YV12")

Spline64Resize(1280, 720)


https://i.imgur.com/2xZHoP8.png

and even more complicated scripts like:


SetMaxCPU("none")
FFVideoSource("\\mibctvan000.avid.mi.bc.sky.it\Ingest\MEDIA\temp\SIC_Preview_REC709_20211124_de.mov")

DeBilinearResizeMT(720, 480)
t = QTGMC( Preset="Slower", InputType=2, ProgSADMask=1.0, ShutterBlur=3)
b = QTGMC( Preset="Slower", InputType=3, PrevGlobals="Reuse" )
Repair( t, b, 1 )
mt_convolution("1","1 2 1",chroma="process")
Blur(0.0, 1.58).Blur(0.0, 1.58).Blur(0.0, 1.58).Blur(0.0, 1.58)
dfttest(sigma=64, tbsize=1, lsb_in=false, lsb=false, Y=true, U=true, V=true, opt=0, dither=0)
dfttest(sigma=64, tbsize=1, lsb_in=false, lsb=false, Y=true, U=true, V=true, opt=0, dither=0)
super = MSuper(pel=2, sharp=1)
bv1 = MAnalyse(super, isb = true, delta = 1, overlap=4)
fv1 = MAnalyse(super, isb = false, delta = 1, overlap=4)
bv2 = MAnalyse(super, isb = true, delta = 2, overlap=4)
fv2 = MAnalyse(super, isb = false, delta = 2, overlap=4)
MDegrain2(super,bv1,fv1,bv2,fv2,thSADC=800, thSAD=800)
Spline64ResizeMT(2048, 858)


work just fine as long as there's SetMaxCPU("none")

https://i.imgur.com/3Z16znb.png


Hence confirming Stephen's theory about intrinsics being the cause of the crashes.

I'm gonna stick with Test 32 in all my servers.

[Dogway]

Yes, no problem, I was just confirming this was known. I installed test33 only to update all the filters that used pixel addressing and could benefit also from scale_inputs.

[pinterf]

Found a typo, resizer related. New build is in progress.
EDIT: I see you've found it as well. Sorry for the inconvenience.

[Dogway]

Found a typo, resizer related. New build is in progress.
EDIT: I see you've found it as well. Sorry for the inconvenience.

Thank you.
BTW do you know if CombinePlanes can be optimized?
Currently the old YtoUV() is faster by some 15%, probably same with mergechroma() and mergeluma().

Also StainlessS stated (https://forum.doom9.org/showthread.php?p=1958765#post1958765)that using 'lut' with scaled_inputs (ie. lut over scaled down to 8-bit expression) would be slower than realtime.

[pinterf]

Rebuild.
Avisynth+ 3.7.1 test build 34 (20211208) (https://drive.google.com/uc?export=download&id=1nVYll8WKoHBYOjh53AxPWnZLXxkeIpzi)

[pinterf]

BTW do you know if CombinePlanes can be optimized?
Currently the old YtoUV() is faster by some 15%, probably same with mergechroma() and mergeluma().

Do yo mean the three input clips version, where Y is obtained from the 1st clip for example?
Maybe. The in-clip plane shuffles are optimized if I remember correctly. The other cases make a new empty frame and copy source planes into that. In YtoUV the original Y plane could be kept and only U and V is copied actually.

Also StainlessS stated (https://forum.doom9.org/showthread.php?p=1958765#post1958765)that using 'lut' with scaled_inputs (ie. lut over scaled down to 8-bit expression) would be slower than realtime.
I've seen it, yes, good question, which can only be proved or discarded if I actually implement it. Tempting :)

[Dogway]

Yes, actually I was testing with the following:
Y = ExtractY()
U = ExtractU()
V = ExtractV()
# some per plane filtering
YtoUV(U,V,Y)
# CombinePlanes(Y,U,V,planes="YUV",sample_clip=a)
But probably there are more cases for optimizations.

Thanks for looking into that.

[VoodooFX]

Can CombinePlanes combine directly to YV12 when U and V clips are half resolution of Y?