AviSynth+ thread Vol.2

[jpsdr]

Big
Now i can build the .lib, and build the avs plugin with llvm. Because of course, building with llvm but using the .lib created with msvc didn't work.
What surprised me, in a good way, is that the .lib provided by Devil SDK worked with both msvc and llvm.

[StvG]

Originally Posted by jpsdr

Because of course, building with llvm but using the .lib created with msvc didn't work.

Remove "/GL" when building with msvc and you will not have issues to use this lib when building avs with clang-cl.

[jpsdr]

again...
I think the "best" maybe is to be able to build everything with llvm (or the same toolchain for another compiler), but this is good to know.
And this probably answer the fact that i can use the Devil lib with llvm.

[qyot27]

AviSynth+ 3.7.5 has been released. There were some things that required a hotfix (particularly on non-x86 platforms), so the list of changes from 3.7.4->3.7.5 is pretty small.

• CMakeLists.txt: if CMAKE_BUILD_TYPE isn't set, use Release
• Update build.yml, retire unsupported 20.04, add 24.04
• Fix an llvm syntax warning.
• rstdoc: update change log
• Bump to IF v11.1: Fix AVS_Value 64 bit data member declaration for 64 bit non Intel (X86_X64) systems.
• rstdoc: open 3.7.5 change log, add Raspberry pi stuff to posix.
• Fix gcc compilation warning
• Fix: resizer crash for non-intel (e.g. aarch64), regression in 3.7.4
• Fix #434: YtoUV crash, regression since 61d2c9a
• Add pkgconf->cmake detection fallback message for SoundTouch
• avisynth+_arm64.iss: assorted fixups
• plugins/CMakeLists.txt: fix VDubFilter build check
• Remove upx.exe from the source tree

[wonkey_monkey]

[amayra]

Why does AVS+ use a file compressor like UPX anyway ?

[qyot27]

AviSynth 2.5/2.6 were UPX-packed, and simply out of inertia some of my early builds of AviSynth+ were as well. But I stopped doing that literally 10 years ago, and the UPX-packed DevIL.dll that was stored in the source tree was changed to an unpacked one in 2020 because it was a hassle to remember to unpack it before building the installer (and of course, the vendored binaries/libraries don't exist in the source tree now anyway).

[tormento]

Originally Posted by amayra

Why does AVS+ use a file compressor like UPX anyway ?

You can anyway unpack them.

[pinterf]

Thanks, qyot27.

Recently, I spent about ten times more time on a topic than I was originally inteded to, but it's an interesting one. Since qyot27 released an aarch64 Windows version of Avisynth+ (Arm64, specifically Arm64-v8 or v8.4a?), I thought I'd add Neon intrinsic support. (Think of Arm Neon as similar to SSE2 or AVX2 on Intel.)

I read the Arm documentation, and the intrinsics are logical and easy to understand, so it wouldn't take long to convert existing Avisynth code to Neon. Since this seemed straightforward, I decided to do it using real Neon intrinsics instead of the "header" method, which translates existing Intel SIMD to more or less usable Arm SIMD code.

To test on Arm, I need a cross compiler and a hardware and an OS.

I have an i7-11700 with plenty of processing power and RAM, so I tried installing Win11 under QEMU. Unfortunately, the how-to guides on the Internet were made before Win11 24H2, which closed the infamous TPM2.0 registry cheat method of installation. I went back to a previous Win11 23H2, which I successfully installed under QEMU. However, it was ridiculously slow - remember, this is real emulation - so slow that I couldn't even start a command prompt. By the time the start menu or search box appeared, a timeout occurred, and the menu collapsed back. I both had a system and not.

Fortunately, my sons bought a Raspberry Pi 5. One is using it for growing grapefruit seeds and monitoring soil humidity , while the other is engaged in web development, so I borrowed his Pi5 for my Avisynth needs. Yes, real aarch64 hardware, no emulation, and with quite usable compilation speed. I successfully built ffmpeg and Avisynth 3.7.4 and started some tests.

Soon, some configuration issues were discovered in the build process, which were fixed and clarified in 3.7.5 in the build environment (See qyot27's changes, fixing default Release optimization when e.g. Ninja was used).

First, I decided to make our resampler code in Avisynth - the C version -more compiler-friendly (and finish some optimization tasks that were postponed in the 3.7.4 release). Vectorizing friendly C code means that even from well-written C, a good compiler would generate very quick SIMD instructions. This is what I've been doing in past weeks: modify, test, benchmark. First on Windows, then on Raspberry Pi 5 Arm64.

Now I can say that MSVC is not capable of any optimization on integer values (e.g., 8-16 bit pixel values) from C code. I tried to help MSVC by writing code that would assist it, but in vain. After a week, I reached the point where I don't want to fight anymore; I cannot win the battle against the compiler's limitations and ignorance.

What works in MSVC's own C++ compiler:

What is already written in direct Intel SIMD code.
- 32-bit float arrays. Maybe. Such operations would be realized and compiled into SIMD code.
- Use MSVC when you need to support ancient OSs.

Conclusion: I'd ban Windows MSVC Microsoft compiler builds for today's processors in use cases where we process integer data and no handcrafted SIMD code is not available.

What to use instead? MSVC + clang-cl or LLVM is perfectly fine. Also, Intel ICX, which is LLVM-based as well. gcc on non-Windows is fine as well.

For Intel/AMD64 processor code, LLVM wins over GCC (as of 2025) by seeing the code it generates (Godbolt Compiler Explorer is a great tool!). The GCC compiler is good, but it's not on LLVM's level. LLVM detects and utilizes many more tricks than any of the others, which surprised me.


And now, some benchmarks.

Horizontal resapling code compiler (Win64, SSE2, etc means the max processor level, the compiler can utilize), 8, 10 and 16 bit.
ClangCl: what VS2022 supports out of box
Intel: Intel C++ Compiler 2025 (LLVM 19, I think)
Figures are in FPS(bit depth)

 417(8)  463 (10)   360(16)              from C, MSVC SSE2     3.7.4
 597(8) 631(10)  651(16)                from C, Intel SSE4.2  3.7.4 
1183(8) 1193(10)  889(16)              from C, Intel AVX2    3.7.4

   91(8)   83(10)    56(16)              from C, MSVC SSE2     3.7.6
 477(8)  468(10) 292(16)                from C, ClangCl SSE2  3.7.6
 429(8)  468(10) 1327(16)(!)           from C, Intel SSE2    3.7.6
1734(8) 2545(10) 1580(16)           from C, Intel SSE4.2  3.7.6
2173(8) 2620(10) 1804(16)        from C, Intel AVX2    3.7.6

3800(8)                                     from SIMD, MSVC AVX2
2870(8)                                     from SIMD, Intel C++ 2025, SSSE3
4370(8)                                     from SIMD, Intel C++ 2025, AVX2

The figures show:
- The C code can be written so as a good compiler can turn it easily to SIMD code.
- MSVC own compiler: no comment
- When the compiler can utilize a more modern processor instruction set (like we build it for "native" arch), speed benefits a lot.

[DTL]

It is also good to test with different frame sizes (relative to typical CPU cache size of about 2..20 MBytes ?) and multiplied to threads number as we have frame-based multithreading in current AVS+. Like with 3 different frame size:

1. Very small - buffer size much less of 2 MBytes.
2. Medium - buffer size is about 2..8 MBytes.
3. Big - buffer size is significantly more of 2..8 MBytes.

Also as current resizer is dual-1D it is good to test H+V changing of frame size (like 2x width and 2x height). Big enough number of tests may be required.

Most benefit of the 'good from-C' compiler expected on poorly SIMD-optimized parts of software as we have less and less human resources for SIMD optimizing and expected AI compilers may still helps somehow.

[wonkey_monkey]

What (if anything) is considered the correct way to ask a user for a colourspace as a filter parameter? Expect a string and convert with ColorSpaceNameToPixelType (or it's C++ equivalent)?

Could/should pixel type numbers be exposed as script function constants?

PS Has something happened to AVSMap recently? I was trying to compile my program after installing 3.7.5 and it was giving me unresolved external errors (I think, might have something different, but it was definitely linking-related), but a rebuild seem to fix it.

[qyot27]

Getting the macOS builds ready for 3.7.4 stalled so long (as a side effect of my access to an Intel version of macOS not working too well this time around, and waffling about where to draw the cutoff for the minimum OS version) that 3.7.5 ended up getting released first. It's probably just as well, since that ARM64 resizer issue that got fixed in 3.7.5 probably also affected macOS.

But I just fixed that issue by uploading the macOS build for 3.7.5. Unlike previous installers for macOS, this time, we have a Universal Binary, so there aren't separate installers for Intel vs. Apple Silicon. The minimum OS version is also set to macOS 13 Ventura, as that is the oldest version still supported by Apple. While it does limit the range of Intel Macs that can be installed to, there is always OpenCore-Legacy-Patcher or you could still build the binaries yourself (and given the differences introduced in 10.15 Catalina, if you're using anything older than that, you'd probably be better served by building it yourself). Unless something drastic has regressed lately, it's still possible to build AviSynth+ for Tiger (10.4) and Leopard (10.5) on PowerPC if you have the patience to let Tigerbrew compile much newer versions of the build tools (CMake, GCC, etc.), so...

Also, between having to disable Gatekeeper to run stuff that isn't code-signed on Apple Silicon (unless, like I mentioned above, you built it yourself) and the shift toward using RPATH shenanigans, the optimal long-term solution for native macOS support for AviSynth+ as well as client applications that use it will be for a package to get added to MacPorts and Homebrew, as they can deal with that weirdness without the user necessarily having to worry about it.

[PoeBear]

Is there a world where the comment and the backslash system could co-exist, similar to the "bug report"/non-working example from the documentation?

ColorBars
ShowFrameNumber
Trim(0,9) # select some frames \
+ Trim(20,29)

I often do similar backslash uses when working with multiple Trims, often from multiple sources. Sometimes it's nice to throw in a comment on each Trim to remind myself why its switching sources. Currently, if I really need a comment, I have to section the Trims it into different clips, then merge them, eg:

ColorBars
ShowFrameNumber
scene1=last.Trim(0,9) #main color footage
scene2=last.Trim(20,21).Grayscale() #black and white scene, remote chroma
scene3=last.Trim(20,29) #back to main color footage
scene1+scene2+scene3

which can get tedious if there's a lot of clips. A long project with multiple sources can get into the many dozens. The more clips, the more chance to get confused by all the Trims and why they exist lol. Would love some 3rd type of comment that is lower precedence than the backslash. It would keep scripts shorter/cleaner, and help when working in the middle of the footage, eg inserting a new Trim without renaming each of the following clips

[wonkey_monkey]

Originally Posted by PoeBear

Is there a world where the comment and the backslash system could co-exist, similar to the "bug report"/non-working example from the documentation?

Can I interest you in ShebangMod and the associated ScriptMonkey plugin? Because I couldn't interest anyone else You can put comments wherever you like and you might not even need backslashes. I can't remember if it allows chaining to cross lines but I can look into adding that.

The current version is forked from AviSynth 3.7.2 but I can compile it again, or you can apply the small changes yourself to the latest source code.

[Gavino]

Originally Posted by PoeBear

Is there a world where the comment and the backslash system could co-exist, similar to the "bug report"/non-working example from the documentation?

Yes, just use the alternative comment form /* ... */, so the non-working example becomes the working:

ColorBars
ShowFrameNumber
Trim(0,9) /* select some frames */ \
+ Trim(20,29)

[qyot27]

Even though Gavino beat me to it, I have another point to make:

Originally Posted by PoeBear

Is there a world where the comment and the backslash system could co-exist, similar to the "bug report"/non-working example from the documentation?



I often do similar backslash uses when working with multiple Trims, often from multiple sources. Sometimes it's nice to throw in a comment on each Trim to remind myself why its switching sources. Currently, if I really need a comment, I have to section the Trims it into different clips, then merge them, eg:



which can get tedious if there's a lot of clips. A long project with multiple sources can get into the many dozens. The more clips, the more chance to get confused by all the Trims and why they exist lol. Would love some 3rd type of comment that is lower precedence than the backslash. It would keep scripts shorter/cleaner, and help when working in the middle of the footage, eg inserting a new Trim without renaming each of the following clips

# isn't the only comment type in AviSynth. C-style comments /* */ are also valid (as well as [* *] for nested comments), and do work in the above example, and in other contexts¹. Essentially, # works like C++-style // comments, and the issue to me seems less about the backslash than it is about how implicit last works when using the splice shorthand syntax. The way others got around it (as per the examples in the Github issue and the old forum posts) was by putting the backslash at the start of the second line, rather than at the end of the first.

¹Like commenting out a line when initializing an array variable that spans multiple lines. # fails entirely for this, you have to use /**/, and don't include the backslash in the comment.

The bug report the documentation mentions seems to have originated here:
https://github.com/AviSynth/AviSynthPlus/issues/6
which links back to
https://forum.doom9.org/showthread.p...41#post1651141
and
https://forum.doom9.org/showpost.php...&postcount=293



The more concerning thing that I can see here is that there is something clearly wrong about the Trim operation in that example:

ColorBars
ShowFrameNumber
Trim(0,9) # select some frames  \
  + Trim(20,29)

As copied directly from the documentation, this produces 1 frame.

 avs2yuv -raw test.avs -o /dev/null
Warning: output will not contain any headers!
You will have to point resolution, framerate and format (and duration) manually to your reading software.
Avs2YUV 0.30
Script file:	test.avs
Resolution:	640x480
Frames per sec:	30000/1001 (29.970)
Total frames:	1
Progress       Frames       FPS   Elapsed    Remain
[  0.0%]      0/1          0.00   0:00:00   0:00:00   
Started:	Wed Apr 30 13:13:52 2025
Finished:	Wed Apr 30 13:13:52 2025
Elapsed:	0:00:00

while

ColorBars
ShowFrameNumber
Trim(0,9) /* select some frames */ \
  + Trim(20,29)

produces 20, as the Trim commands would imply:

$ avs2yuv -raw test.avs -o /dev/null
Warning: output will not contain any headers!
You will have to point resolution, framerate and format (and duration) manually to your reading software.
Avs2YUV 0.30
Script file:	test.avs
Resolution:	640x480
Frames per sec:	30000/1001 (29.970)
Total frames:	20
Progress       Frames       FPS   Elapsed    Remain
[ 95.0%]     19/20      1900.00   0:00:00   0:00:00   
Started:	Wed Apr 30 13:15:41 2025
Finished:	Wed Apr 30 13:15:41 2025
Elapsed:	0:00:00

The implicit last is being set to *something*, and the Trim(20,29) command is appending to it, but it's all getting trashed down to a single frame if the backslash before the splice is commented out by a #. The documentation says that it's not returning [0..9,20..29], which is correct, but it's also not returning either one of [0..9] or [20..29] (or [20..29] appended to...whatever last was set to).



Secondarily, and going back to the point I made above about the arrays, is there any reason why the multi-line syntax needs to work the way it does when enclosed by any of the grouping syntax elements?

It's been a while, but I don't recall whether the {} in the function syntax require backslashes to allow lines to be spanned, because function definitions work more like C than typical script, and you'd only need the multi-line spanning in a case where the command inside the function would normally need it, but not the function as a whole.

And again, if you're initializing an array, I don't see why we'd need backslashes between the [] if it's just counting elements between those two characters. Is there some alternate use of these things that means that usage is ambiguous and multi-line span with backslash *needs* to still apply to them?

[wonkey_monkey]

Implicit last in this case is, I think, Trim(0,9). I would guess that "+" is inventing its own 0-frame clip for the LHS (based on the RHS?), since that is empty, then trim(20,29) acts on implicit last ("Trim(0,9)") - but since implicit last doesn't have frames 20-29, it just returns the last frame of it, which is frame 9.

Edit: actually it probably makes more sense that the "+" is just nulling itself in parsing: [null] + x => x

[qyot27]

Originally Posted by wonkey_monkey

Edit: actually it probably makes more sense that the "+" is just nulling itself in parsing: [null] + x => x

That does seem to be exactly what it's doing, which is leading to Trim(20,29) returning frame 9 instead. I don't think that's* expected behavior, but I could be wrong.

*Trimming a frame range that doesn't exist, I mean. -1 obviously has a special meaning, but a positive range that lies outside the valid already-trimmed range? Or was this the same thing that people complained about last time because it was an anti-feature they relied on?



So I guess here's a different question: within the parser, are comments removed first (as a sort of dead code elimination, the way a compiler would do it) before being executed? And if it does, and you comment a line, is the line blanked, or is it removed? Meaning: the newline is collapsed so that when you comment out line 2, does line 3 become line 2 or does it stay line 3 and line 2 still exists but empty? Practical experience seems to point toward the latter, since you can't comment out a line in the middle of a multi-line array initializer, even though most people would think it should collapse and be treated as if the line was deleted entirely.

[wonkey_monkey]

Well you can't do this:

ColorBars(\
width = 1024,\
/* asduhsd */
height = 1280)

or

ColorBars(\
width = 1024,\
# asduhsd
height = 1280)

So it seems it doesn't remove newlines near comments.

I would say AviSynth syntax is a tad outdated, hence my efforts to get around its limitations with ShebangMod. Just changing it outright would be too disruptive, especially when you've got things like ScriptClip to take into account, but an opt-in with a special header line seems like a good way to go (e.g. "#!AviSynth4"). Perhaps one simple modernisation could be "If you encounter a parsing error due to a newline, remove the newline and try again"?

Though that wouldn't help in the + Trim case above...

In Javascript you can do:

x = 6
+4
alert(x)

and it will alert "10". So maybe that works the opposite way - if you get a parsing error, assume an end of statement and try again. You can also force and end of statement with ";" of course, so:

x = 6;
+4
alert(x)

comes back as "6".

[Gavino]

Originally Posted by qyot27

That does seem to be exactly what it's doing, which is leading to Trim(20,29) returning frame 9 instead. I don't think that's* expected behavior, but I could be wrong.

*Trimming a frame range that doesn't exist, I mean. -1 obviously has a special meaning, but a positive range that lies outside the valid already-trimmed range? Or was this the same thing that people complained about last time because it was an anti-feature they relied on?

So I guess here's a different question: within the parser, are comments removed first (as a sort of dead code elimination, the way a compiler would do it) before being executed? And if it does, and you comment a line, is the line blanked, or is it removed? Meaning: the newline is collapsed so that when you comment out line 2, does line 3 become line 2 or does it stay line 3 and line 2 still exists but empty? Practical experience seems to point toward the latter, since you can't comment out a line in the middle of a multi-line array initializer, even though most people would think it should collapse and be treated as if the line was deleted entirely.

Yes, a unary '+' is simply treated as a null-op (even when applied to a clip, which doesn't make much sense).

And Trim clamps its frame arguments to the frame range of the source clip, so +Trim(20,29) applied to a 10-frame clip becomes, in effect, Trim(9,9).

Commenting out an entire line does not make the line disappear, since the new line at its end is not part of the comment!
(unless of course it's inside a block comment that has not yet been closed and thus extends onto the next line...)