Avisynth+

[ultim]

[MODERATOR EDIT:]
Current branch as of November 2019:
The pinterf fork was integrated (https://forum.doom9.org/showthread.php?p=1888102#post1888102) in main line like AviSynth+ 3.4.0
Download and sources: From GitHub (https://github.com/AviSynth/AviSynthPlus/releases)

Older superseded branch/fork:
[Download: From GitHub (https://github.com/pinterf/AviSynthPlus/releases) (use version with vc_redist when installing for the first time)
Sources: https://github.com/pinterf/AviSynthPlus]
[END MODERATOR EDIT]

Avisynth+ compared to Avisynth 2.6
- 64-bit version compatible
- Multithreaded
- High bit-depth support
- Faster script startup and substantially faster core filters
- Support for multiple ("shadow") plugin directories
- Autoloading of C-plugins
- Script language extensions, with support for multiline conditionals and loops.
- Improved still image support and TimeStretch
… and more to come as development continues

Avisynth+ features for developers
- Easy and fast compilation, with current compilers
- 2x increased compilation speed
- Improved debugging experience
- Leaner and more logical project structure
- Large parts of the code refactored
… and being continuously improved

How to install
Use installer for the latest release from here (https://github.com/AviSynth/AviSynthPlus/releases). If classic AviSynth is already installed, you will be offered the choice of either upgrading or replacing it. In the first case you'll be able to switch back to classic AviSynth by simply uninstalling AviSynth+. Your existing plugins will be kept in both cases.

Can I just replace the old avisynth.dll with the new one from AviSynth+?
Yes, with some caveats. In Avisynth+ some core plugins have been moved to external plugin DLLs (but are still shipped with the core), so you'll manually need to copy those too. Avisynth+ uses a different C++ runtime, which needs to be installed separately. And last but not least, this will make your Avisynth and Avisynth+ plugins mix in the same folders, which you might not want (though it is harmless ATM). All in all, I recommend you use the Avisynth+ installer instead, which will take care of all these problems for you.

Compatiblity to Avisynth
Avisynth+ tries to provide a superset of Avisynth's features while staying compatible to existing code. This means you should be able to use plugins and scripts written for Avisynth without any problems with Avisynth+. The contrary however might not be true: Plugins and scripts written explicitly to take advantage of Avisynth+ might not work using the official Avisynth.

Shortcomings compared to Avisynth
- Support for Windows 98/Me/2000 dropped. Oldest supported OS in Windows XP.
- TCPDeliver not included in the sources. However, TCPDeliver from official Avisynth can be used with Avisynth+.
- Ability to load VFAPI filters is missing, hopefully temporarily.

How to compile (without DirectShowSource)
- Get Git for Windows (https://git-scm.com/download/win).
- Get CMake (http://www.cmake.org/cmake/resources/software.html).
- Get Visual Studio 2013 Community Version (https://www.visualstudio.com/en-us/news/vs2013-community-vs.aspx).
- Clone the repository (https://github.com/pylorak/avisynth) from GitHub.
- Configure the project using CMake. No manual changes to CMake variables are necessary, just hit Configure, select yout compiler then Generate.
- Load generated VC++ solution and compile.
- Done.
- OR alternatively to the above steps: For people who prefer to compile from the command line, qyot27 was kind enough to provide detailed compilation instructions from an Msys environment. See http://forum.doom9.org/showthread.php?p=1643929#post1643929 .

This is a community project and you are welcome to work on improving Avisynth+. Let us know what you're working on either here or on #avs-plus on Rizon, and join the discussions. Feel free to work on any feature you'd like, or if you are feeling unsure where to start, choose an open issue from the GitHub page. Testing, documenting, and giving any kind of feedback is also a great help.

Avisynth Plugin Writing Tips Series
Tip #1: Exceptions (http://forum.doom9.org/showthread.php?p=1647262#post1647262)
Tip #2: Parallel execution (http://forum.doom9.org/showthread.php?p=1649886#post1649886)
Tip #3: Choosing your AviSynth header (http://forum.doom9.org/showpost.php?p=1712979&postcount=938)

[Groucho2004]

Feedback?
A couple of observations after running a bunch of my test scripts:
Exception handling is different than the official (and my ICL) DLL. When Avisynth runs out of memory, this dialog is displayed:
http://www.iol.ie/~schubert/crash.png

The official version(s) properly throw an error message like this:
GetFrameBuffer: Returned a VFB with a 0 data pointer!
size=12912704, max=2109149184, used=1564790336
I think we have run out of memory folks!

The performance is about the same as with the official DLL, sometimes a little bit better. The ICL DLL is in some rare cases much faster.
However, this is rather academic since the plugins and filters determine the performance by a large margin.

You should mention that the DLL is dynamically linked and that one needs the VC2012 runtime DLLs. I had to figure this out with Dependency Walker.

All this was tested on XP32, SP3.

[burfadel]

Both static and dynamically loaded libraries have their advantages and disadvantages. Static libraries give the benefit that you don't need to have other stuff installed, but the downside is that it will most likely be an outdated version of that library (say, VS2010). The same goes with programs with the VC++ libraries in their folder, the program will use that version instead of the one on the system. The real issue with this is that newer versions of the runtime libraries fix bugs and improve stability, possibly improve performance, and close security holes. Also having things static and the individual files means using space on the hard drive that really isn't required. Since the VC++ runtime libraries are basically universally required there should be no need to have the libraries statically loaded.

Most systems don't have the latest runtimes loaded and (unless you delete them, which is 99 percent safe to do so) have old versions of the libraries in varies program folders.

Anyways, so far testing that build I haven't had any issues.

[StainlessS]

If it dont compile well with later version compiler, then have a go at them.
VS6 is just fine and any later problems come from later lack of compatibility.
Really annoys me when you need the latest version IE v...whatever to view a web page, misguided at the very least.
I'm sure that you mean well but it dont need that kind of upheaval.

Several posts preceded my outburst, was aimed at Ultim.

[TheFluff]

VS6 is just fine and any later problems come from later lack of compatibility.

this statement is so bizarrely wrong that I am unable to formulate a polite response to it

[StainlessS]

this statement is so bizarrely wrong that I am unable to formulate a polite response to it

Not much different there then. :)

[qyot27]

If for no other reason, this certainly makes the sources dead simple to build for those of us used to cross-compiling.

Ah, what the heck. Build instructions suitable for copy-and-paste:



This guide assumes a familiarity with a more *nix-centric command line compilation methodology, because it's easier to provide direct instructions for this that can just be copy/pasted.

MSys and 7zip should already be installed, and the user should know what a phrase like 'adding the directory to the system PATH' means.




[AviSynth+ prerequisites]
r1576 is the last milestone version that can built on XP with VS2010.
Newer versions (which is what we care about) need at least VS2012. XP
compatibility is still do-able by adjusting the toolset CMake uses.

Install CMake:
http://www.cmake.org/cmake/resources/software.html

Add CMake's directory to the system PATH manually if the installer won't.

Add 7zip to the PATH. Make sure that MSys' root directory (the
one holding msys.bat) is also on the PATH.
[DirectShowSource Prerequisites]

DirectShowSource requires a couple of big pieces here that building the AviSynth+ core itself does not. It's not a requirement to build DirectShowSource, especially with the options of using either FFmpegSource2 or LSMASHSource, but the guide wouldn't be complete if I didn't cover it.

DirectShowSource requires strmbase.lib, the C++ Base Classes library, which for some reason isn't included in a standard install of Visual Studio. The source code for the library is provided with the Windows SDK, and requires the user to build it first.

Download the Windows SDK 7.1:
http://www.microsoft.com/en-US/download/details.aspx?Id=8442

Download the following ISO for x86:
GRMSDK_EN_DVD.iso

Verify against CRC32 or SHA1:
CRC#: 0xBD8F1237
SHA1: 0xCDE254E83677C34C8FD509D6B733C32002FE3572

For convenience, you can use Pismo File Mount (if you've already got it installed for AVFS)
to mount the ISO. Then just launch setup.exe and follow the wizard.

Install only the Samples, uncheck everything else. Intellisense stays checked, though.

Open VS2013 Express (or what is it, Community now?), and open the .sln file in the 7.1 SDK, at
C:\Program Files\Microsoft SDKs\Windows\v7.1\Samples\multimedia\directshow\baseclasses

Allow VS2013 to convert the project, switch it over to Release, and Build.
That's it. For pre-SSE2 CPUs, make sure you switch the ARCH to either IA32 or SSE rather
than leaving it disabled. This is buried in the Properties dialog.
DirectShowSource also needs the DirectX SDK, so we'll grab that too.
Download the DirectX SDK (February 2010):
http://www.microsoft.com/en-us/download/details.aspx?id=10084

Only install the headers and libs from it. Uncheck everything else. Unlike the
strmbase.lib stuff, there's no post-install stuff required.

To make the AviSynth+ build instructions more concise, we'll set a couple of environment variables. Open up MSys' /etc/profile and copy the following two lines into it somewhere:
export STRMBASELIB="C:/Program Files/Microsoft SDKs/Windows/v7.1/Samples/multimedia/directshow/baseclasses/Release/strmbase.lib"
export DIRECTXSDK="C:/Program Files/Microsoft DirectX SDK (February 2010)/Include"
Thankfully, all of this setup only needs to be done once.


[Building AviSynth+]

Start the Visual Studio Command Prompt

You can use Visual Studio's stuff from MSys by launching MSys from the Visual Studio Command Prompt. So type 'msys' and hit Enter.

Download the AviSynth+ source:
git clone git://github.com/AviSynth/AviSynthPlus.git
mkdir avisynth_build
cd AviSynthPlus
Set up the packaging directory for later:
AVSDIRNAME=avisynth+_r$(git rev-list --count HEAD)-$(date --rfc-3339=date | sed 's/-//g')
mkdir ../$AVSDIRNAME
If DirectShowSource won't be built, comment out the reference to its subdirectory in plugins/CMakeLists.txt:
sed -i '4 s/^/#/' plugins/CMakeLists.txt
Now, build AviSynth+. We'll do this out-of-tree so it's easy to clean up afterward:

cd ../avisynth_build

For 32-bit:

cmake -G "Visual Studio 12 2013" -T "v120_xp" -DDSHOWSRC_BASECLASSES_LIB="$STRMBASELIB" -DDSHOWSRC_DX_INCLUDE_PATH="$DIRECTXSDK"
cmake --build . --config release
mv Output avsplus_x86
mv avsplus_x86 ../$AVSDIRNAME
rm -R *

For 64-bit (DirectShowSource isn't really worth the effort on 64-bit, and most 64-bit users won't be on XP Pro 64-bit):

cmake -G "Visual Studio 12 2013 Win64"
cmake --build . --config release
mv Output avsplus_x64
mv avsplus_x64 ../$AVSDIRNAME
rm -R *
cd ../

You can't build 32-bit and 64-bit at the same time.

If you decided not to build DirectShowSource, omit the -DDSHOWSRC_BASECLASSES_LIB and -DDSHOWSRC_DX_INCLUDE_PATH options from the (32-bit) cmake command above.

Packing the newly-built DLLs with 7zip:
7za a -mx9 $AVSDIRNAME.7z $AVSDIRNAME

[ultim]

A couple of observations after running a bunch of my test scripts:
Exception handling is different than the official (and my ICL) DLL.
...
You should mention that the DLL is dynamically linked and that one needs the VC2012 runtime DLLs. I had to figure this out with Dependency Walker.

All this was tested on XP32, SP3.

Thanks, later today (after work) I'll reupload a version built with VS2005 to decrease the probability of VC runtimes needing to be installed. I'll also take a look at the exception handling problem.

The performance is about the same as with the official DLL, sometimes a little bit better. The ICL DLL is in some rare cases much faster.
That is about expected. As I mentioned in my original post, the optimizations are mostly to the initialization path. During frame-processing most time is spent in the plugins, BitBlt, and video frame allocation. I wasn't able to make BitBlt conclusively faster for now, and I've yet to take a closer look at the frame allocation path. Either way, most time is still spent in the plugins, so the only way to really speed Avisynth up is to improve the plugins.

If for no other reason, this certainly makes the sources dead simple to build for those of us used to cross-compiling.
Making it easy to build was the primary goal for now after all. Thanks for the additional instructions too. But as a note to others, you don't need the command line to compile if you are not familiar with it. Using CMake GUI and the VS IDE allows you to accomplish everything, even disabling the build of DirectShowSource (just unload the project in the IDE).

What would be great is if we could start working on making the AVS core 32/64-bit safe.
That is one of my long-term goals, it will probably take a lot of time though and I'm not working on that directly right now.

[ultim]

If it dont compile well with later version compiler, then have a go at them.
VS6 is just fine and any later problems come from later lack of compatibility.
Really annoys me when you need the latest version IE v...whatever to view a web page, misguided at the very least.
I'm sure that you mean well but it dont need that kind of upheaval.

Several posts preceded my outburst, was aimed at Ultim.

Hehe, I don't think I've made an upheaval, but if I did I'm sorry, I didn't intend to. But where I think you're wrong is that you are neglecting most of the community. Certainly, Avisynth doesn't strictly "need" a VS newer than VS6 to compile: compiled with VS6 it will work fine (not too surprising given it was mostly developed on VS6). But the people need newer VS versions. VS6 is so old that most people are used to newer environments, which by the way also brings better features for both debugging and comfort, as well as better compatibility for new OSs.

I find your comparison to a webpage needing the latest browser version highly flawed. Needing only a newer version just for the sake of upgrading sure is misguided. But I didn't upgrade the project to compile only with VS2013. I left VS2005 (which is 8 years old!) and anything newer than that fully supported, and I upgraded for the sake of making it easier for people to compile. That makes 4 different Visual Studio generations supported! If you really want an analogon to webpages, I'd say my move is more like MS trying to banish IE6, which everyone is happy about, except for people who still develop for IE6.

If you intend to keep Avisynth stuck to VS6, it will be an entry barrier for many people who'd otherwise try to contribute to the project. So this is why I think compatibility with VS2005+ is more important than VS6.

[Groucho2004]

You should mention that the DLL is dynamically linked and that one needs the VC2012 runtime DLLs.
Thanks, later today (after work) I'll reupload a version built with VS2005 to decrease the probability of VC runtimes needing to be installed.
I only mentioned this so people know that the VS2012 runtimes are a requirement.

[IanB]

It's obvious you have done a lot of work and are quite keen. You say you want your changes accepted upstream, yet you randomise the location of files and you make copious white changes to the sources. Hell, I would have love to restructure the tree to the way I think it should be, but I resist that urge very strongly. If you want to contribute changes you need to make them easy, very easy, extremely very easy to incorporate back into the original tree. If I download and apply a set of patches and compare the changes and see a small handful of very neat, very tidy and very concise changes, I will probably show a fair amount of interest and actively review the changes, checking the boundary conditions, inspecting the performance and considering the side effects. If I download a set of patches and they won't even apply because the files and directories have been moved and/or renamed, I probably won't bother looking any further. And if I do look further and I find a pant load of spurious white changes hiding the important differences then almost certainly I won't bother investigating further.

[ultim]

It is true that there is a lot of noise. However, I disagree that they hide the useful differences, because I always kept them separate. For example, any time I moved files I did that in a separate commit without changing contents, so that Git can clearly tell that it was only a rename, and I only changed the files in the next commit. Never in the same commit so that Git recognize moved files and not flag them as removed/added.

Also, important commits which fix bugs or affect well-defined features are also committed separately, to prevent them from mixing with "unimportant" changes. I was under the impression that you are also using Git, because your latest changes are in Git on GitHub. If you are in fact using Git, my changes will apply cleanly if you merge from my repository, and not export/import patchfiles. And even if you are using SVN, there are working bridges between Git and SVN so you can still pull my changes.

So I don't really see your problem, especially because I *did* keep noise and useful changes in separate commits.

Edit: Btw, telling me that I have randomized the file locations went a bit overboard. Anyone who checks out my repository can tell that the project structure is a thousand times cleaner than it ever was in the history of Avisynth.

[Groucho2004]

I was under the impression that you are also using Git, because your latest changes are in Git on GitHub.
Avisynth is hosted here (http://avisynth2.cvs.sourceforge.net/viewvc/avisynth2/) and as far as I know, Ian is only maintaining it using CVS.

[ultim]

So Avisynth is only accepting contributions in the form of exported patch files? I have a full repository with cleanly separated changes and fully preserved history, and it is termed useless?

[TheFluff]

So Avisynth is only accepting contributions in the form of exported patch files? I have a full repository with cleanly separated changes and fully preserved history, and it is termed useless?

Avisynth is in many ways a project that is permanently stuck in the past, and all efforts have tried to modernize it are either forks (Avisynth-MT, Avisynth 64) or completely different projects (Avisynth 3, VapourSynth). You've found the reason for this the hard way.

Avisynth 2.6 is still a late 90's project at its core. Even when someone finally got around to breaking the API (after ten years), high bitdepth video still isn't supported, nor is 64-bit compilation (there are even a few comments in avisynth.h going "this will break on 64-bit!", gee thanks, it's almost 2014 now), all the old braindamaged API restrictions that sorta made sense in 2001 are still there and you still need to use CVS and support VC6, Windows 98 and pre-SSE CPU's because that's how it's always worked. Nobody is interested in developer friendly changes because nobody is developing Avisynth except poor IanB, and he understandably doesn't want to change his own environment.

If you want to get anything done, I suggest you either fork or start contributing to VapourSynth.

[mandarinka]

Thanks for contributing to Avisynth! You really made it hard to merge upstream though, since you have a lot of changes at once and with the project files reshuffling, it is sort of like a "code dump" for the upstream/original developers.

Ideally, the changes should be reviewed and then adopted, but in this form, it will be hard to do...

[ultim]

Thanks for contributing to Avisynth! You really made it hard to merge upstream though, since you have a lot of changes at once and with the project files reshuffling, it is sort of like a "code dump" for the upstream/original developers.

Ideally, the changes should be reviewed and then adopted, but in this form, it will be hard to do...

I understand that my "patches" are a lot of code. However, nobody is forcing IanB to review and merge them in one night. My commits are on GitHub and they won't go anywhere. He could take them one-by-one, and review and merge them over a course of 2-3 weeks or more. Just because I published them all at once does not mean they need to be reviewed all at once, especially because they are broken up into lots of small commits, with only a few exceptions towards the beginning.

EDIT: I'd like to make it clear just once more, that me shuffling files should not make it that much harder to incorporate the changes. Because I have always done the "moving files" and "changing contents" in separate commits. So if upstream ever comes to a commit of mine which moved files, all he has to do is move files. Anything that changes contents will show up as a new commit, so there is never a case of a change being hidden just because the whole file shows up as new falsely. There is no friendlier or more respecting way to move files than this (except for the case when Ian moved to Git or SVN, in which case he wouldn't even have to move the files by hand).

... and all efforts have tried to modernize it are either forks (Avisynth-MT, Avisynth 64) or completely different projects (Avisynth 3, VapourSynth). ...
If you want to get anything done, I suggest you either fork or start contributing to VapourSynth.

Well, I guess it is time for me to present my full story. I have known Avisynth for a very long time, but until about 8-9 months ago, I only new about some of its forks. So I've began laying out plans how I could implement a better Avisynth, because no matter how much I liked Avisynth, its flaws did (and do) bother me. Off the top of my head my ideas were: built for multithreading in its core, video frames should be taggable by custom information, 64-bit compatbility, cross-platform support, just to name a few. In one word: very (too?) ambitious. I had absolutely no time to implement this though until a few months passed. About half a year ago then, I started laying out an improved .avs scripting language, with full support for old scripts, but better language features, like proper multiline if-blocks, multiple return values, and some more. Actually, that was the sole reason I ported the Gold Parser to C++ (https://github.com/pylorak/GoldCPP). But I've done more than just port Gold Parser, I've even written and tested its language definition input file for my improved .avs syntax, in correct EBNF(-like) form and being able to parse!

Then, about this time, a "strange" (LOL) idea struck me: What if others have also started to make something that I was just planning to? So I fired up Google, and have found VapourSynth and Avxsynth. I saw that VapourSynth set mostly the same goals as I did, which was cool, but I was *very* annoyed by its choice of Python. Yes, it is a more powerful language, but also much harder to learn and use, and the syntax as implemented in VapourSynth is just too verbose. Anyway, I decided it is better if I help out one of the projects instead of just starting yet a another fork/alternative, and my decision fell to Avisynth. So after yet another few months of doing-other-things (diploma thesis, some free software, and looking for work), here I am contributing to Avisynth.

Btw, I updated my announcement post with a new binary that only requires a much older VC++ runtime. I also looked at the exception handling reported by Groucho2004, and right now my best bet is that the "old" behavior is pure luck. Because even in the old code (and there is no difference to this in my changes), if we run out of memory, new memory must still be reserved for the red error clip (unless there is a frame in the free-list). Now, if there was not enough memory to reserve a new frame, there is also the probability that there won't be enough memory to construct the error clip either -> in which case the code can only crash, both old and new. I have verified my code on countless other exceptions and for me they always correctly resulted in an error clip, and the only time my build crashed was when constructing the error clip failed due to low memory.

[TheFluff]

I saw that VapourSynth set mostly the same goals as I did, which was cool, but I was *very* annoyed by its choice of Python. Yes, it is a more powerful language, but also much harder to learn and use, and the syntax as implemented in VapourSynth is just too verbose.
As Stephen says, VapourSynth isn't tied to its scripting language like Avisynth is. VapourSynth is really just an API; the Python scripting component is just a module that calls that API. You can implement a new VapourSynth API wrapper in any language you want.

[qyot27]

Making it easy to build was the primary goal for now after all. Thanks for the additional instructions too. But as a note to others, you don't need the command line to compile if you are not familiar with it. Using CMake GUI and the VS IDE allows you to accomplish everything, even disabling the build of DirectShowSource (just unload the project in the IDE).
This is true. I just hate having to open up the Visual Studio IDE. Terminal-based compilation has me spoiled.

For that matter, my recommendation of hooking it into MSys isn't actually required either, but I'm not as comfortable with using variables and for loops through cmd.exe (also, the variables and for loops aren't necessary either if the user doesn't plan on using UPX packing or packing the generated .dlls up into a nicely-named archive). It can be done strictly through the Visual Studio Command Prompt.

[ultim]

Obviously, adding .avs syntax to VapourSynth crossed my mind too. The reason I chose Avisynth is because, even though VapourSynth has the same goals that I do, it is not quite there yet. It was built with threading in mind, yet it was (still is?) a gamble whether it locks up with Avisynth filters (and it doesn't have that many native fiilters). It also wants to be portable, but the tools for Linux are still missing (the best advice on the forums to step through a .vps script frame-by-frame is still to use VirtualDub under Wine, LOL). And just like me, VapourSynth also wants to be 64bit-compatible, but it cannot be built for x64 Windows. Furthermore, back then a few months ago, VapourSynth was missing widespread support in application like VDub, a console piper, ffms2 and so on. Plugin support is also larger, coz' AFAIK Vapour can't handle all Avisynth plugins. So as you can see, just because it has better goals than Avisynth, it doesn't mean it is more usable than Avisynth.

Avisynth of course also has most of these problems, yet due to better documentation, better support in applications, and the larger selection of plugins, it is still more usable. And given that I want to use it (not just develop it), these shortcomings played a large role when I've chosen Avisynth to contribute to. I see that some of the mentioned issues have been fixed in r18 and r19, but others are still current and will need a later release to actually fix.

Another, less practical thing that discouraged me from working on VapourSynth as a developer was looking at its sources. Avisynth's code is in terrible shape, I'm not gonna beat around the bush. However, being 10 years old, and having been patched up with functionality that it wasn't designed to handle at all countless times, is at least an acceptable excuse. VapourSynth on the other hand, has similar spaghetti code, is even less documented, and just as hard to read. For a project that set out to correct Avisynth's mistakes, still young and with less features, this was unforgivable in my eyes.

All these things being said, you can see I had more reasons to choose Avisynth than just the scripting language alone.

[ultim]

On a (not-so-) unrelated note, given that Avisynth's development has been mostly dormant with only very few contributors, it is not very motivating when somebody steps up with work and willingness to make the situation better, to tell him "go and work on that other project". Especially because nobody has said anything wrong about my proposals. If it had problems like breaking existing functionality or making code worse I'd understand. Or if IanB had reviewed them and told me "you should fix that differently because...", or any kind of constructive criticism, I'd go back and fix those things ASAP. But literally the only problem some of you had with my changes is that "there are too many", which makes me LOL in sadness. Should I beg for forgiveness that I contribute to the project at all?

[MasterNobody]

ultim
I think problem with review is not that you have too many functional changes but due your move/rename files commits. You can't really review/apply patches after it if you not going to apply this file structure refactoring patch (and as I understand IanB don't want to change file structure only because you don't like it). And so you can't cherry pick only ok commits. This way patches have "all or nothing" condition to apply which is not good.

[ultim]

There are multiple solutions to this problem, off the top my head (without any particular order of preference):

IanB could check out my repository in Git without merging any patches, to look at the directory structure I have arrived at. If he doesn't like the placement of some files, he can make suggestions to me how I should rebase the code, and ofc I'd be willing to help.
IanB could at least look at the directory structure I have arrived at. Maybe he'll come to the same conclusion that the new structure really is better than the old one, in which case he can merge the code. Selectively picking commits after the commits that move files (which come at the very beginning) is not a problem any more in this case.
IanB could reorder the files as I did, then selectively pick commits (which will be trivial since he moved the files too), and then move the files back to their original places. He'll need to fix up some #include lines, but this is basically not much work either. But this method allows him to cherry-pick my commits without accepting the new directory structure.


So the only thing left to 'solve' is the manual work needed to actually move the files for the first commit. This can be done fully automatically, even if he is using CVS, because all my work is based on Ian's latest commit. All he needs to do is copy his CVS repo into a new folder, delete the sources without deleting the history, and then check out my Git repo into the same folder. Bang, no need for patches, no need to merge, and no need to move files manually. At this point he can start committing to his CVS if he decides he likes my work. And if he does not like it (after inspecting it), he still has his own repository untouched.

The problem is, is that IanB does not even want to take a look if I have created a logical directory structure. He simply decided that he will not take a look at my changes just because I have moved files, even though they do not hide my changes, because I kept moves from changes separate.

Note that looking at the first commit of mine looks much scarier than it really is. A lot less files have been moved than it looks like, because a large part of the additions come from not moves, but from adding the HTML sources of the FilterSDK to the repository. 99% rest of the moves are whole directory moves, like the 'src' folder being renamed to 'core', or moving the pfc and soundtouch directories into plugin folders. I think you can probably count the number of files I have moved within the core itself on one hand. The rest are as described.

Please, tell me honestly: If you compare the structure of the CVS repository and my Git, isn't the new folder structure much better? Because if it is, then this whole discussion is meaningless. And if it is not better, I have still described two methods in the bulletpoints above, how Ian can fully or partially reject my folder structure while still picking other commits from me.

[Guest]

Here's another option. Make your patches to the existing directory structure.

[ultim]

Here's another option. Make your patches to the existing directory structure.

That falls within my first bulletpoint. And honestly, I'd even be willing to comply (even if not completely happy) if he said, that my directory structure was looked at seriously, but he found it inferior to the original because such and such. I don't think this is an unreasonable request from a contributor, to at least take a look at his work, and only reject it if there are founded objections. This goes for moving files too.

What I am disappointed at, is if Ian decided not to inspect my new structure just because, out of spite. Unfortunately, from his last post it looks like that this is pretty much the case, but I still have some hope left. I hope he'll post soon again to let us know his course of action.

[Guest]

Saying IanB rejects your stuff out of spite is way over the top and won't endear you to our members!

Maybe he doesn't even have GIT, or doesn't have the time to come up to speed on it. Personally, I would have to spend (waste?) many hours just to get to the point of being able to look at your changes.

And you are asking *him* to undo your directory structure changes?! There's some major chutzpah there, no?

[ultim]

Saying IanB rejects your stuff out of spite is way over the top and won't endear you to our members!

Maybe he doesn't even have GIT, or doesn't have the time to come up to speed on it. Personally, I would have to spend (waste?) many hours just to get to the point of being able to look at your changes.

And you are asking *him* to undo your directory structure changes?! There's some major chutzpah there, no?

Please don't stir up emotions. If you read my previous post carefully, you'll see that I have written, "... if Ian decided to...", which is hypothetical, and I ended my post by saying I hope this is not so, and that I hope for the better. I never said that he is doing it out of spite, I said I am hoping he is not.

I also wasn't asking him to undo my changes, it was one of the multiple possibilities, just like the one where he tells me what to change, and I'd do it myself (1st point). Grabbing isolated sentences out of my post and changing the overall meaning is not fair. Again, I'm trying to cooperate here, offering further work to fix what you ask me to, all I ask for is a fair review with arguments. There is no need to install Git or spend hours just to start taking a look as you mention it, because you can browse the source online on GitHub.

So no, no chutzpah.

[Guest]

You said this:

"if Ian decided not to inspect my new structure just because, out of spite. Unfortunately, from his last post it looks like that this is pretty much the case"

That is crystal clear to anyone with a brain.

[ultim]

I also said that "grabbing isolated sentences out of my post and changing the overall meaning is not fair", and you left off the end of the other sentence, which is again, changing the meaning. But since you look like you honestly misunderstood it, I'll say: Sorry. I'm, sorry. I didn't mean it that way, but because you obviously don't believe me, let me apologize.

[ultim]

Please let me try to express myself just once more, one last time, because there was some bad choice of words in the discussion with neuron2 that lead to misunderstandings.

I have made multiple commits over the course of a month, IMHO improving Avisynth's code in many different aspects. When submitting my changes, I ...

published my changes with full history on GitHub,
I based my work on the latest commits by upstream,
I have clearly separated commits,
I moved files in different commits than I made changes, so that changes are not obscured,
I am willing to further patch up my work based on suggestions after a review,
and the code I submit is commented.


Hence, I think that my commits are upstream-friendly, with the worst offender being the file moves. The friction between us is that you tend to view my changes as a bunch of patches, where I, unfortunately, messed up by reorganizing files before the patches. However, I'd like you to look at my file moves as part of the improvements too. I didn't reorganize the folder just because I was feeling like it. They are meant to be part of the improvements, and I feel the new folders make the project structure much more logical, because:

I have moved files and libraries which are part of the core into the core.
I have moved files and libraries which are NOT part of the core out of the core.
I have separated non-essential filters into plugin DLLs.
I have reduced the dependencies on 3rd-party code.
And, by all the above, I also have made it easier for new contributors to find the files they are looking for,
and easier to get a grip on the project.


So, me moving files around is meant to be part of the changes, not some extra stuff that I messed up by chance. And I sure have not "randomized" the locations like IanB put it.

This work took me a month, and as an example, my whole last weekend. In total more than a full day was spent merging, rebasing and cleaning up commits, because there were orignally about 1.5x-2x as many. So yes, I think I've been a good and diligent contributor. The only "bad" things left are the file moves, but I'd prefer not to get rid of them as they ARE part of the improvements, because of the numerous reasons that I have listed above. (And yes, please also note that I can argument why the folder structure is better this way.)

And when I publish my work, people tell me they don't even want to take a peek at any of my work just because they wouldn't have moved files in the first 2-3 commits (which are also improvements), even though it would take them only a fraction of the time that I needed to implement the whole thing. Whereas, if they did, they might even find that the moves make sense.

I hope I managed to bring you closer to my point of view.

[Keiyakusha]

I haven't read the whole stuff, only last post and I don't quite get it. You say "published my changes with full history on GitHub". Well ok, and what do you expect to see after that? If I would be on IanB's place I wouldn't even move my ass unless you'll send me patch for my own repo that I can review and apply. Maybe if avisynth was a super-active project with some devs that have tons of motivation... But that's not the case.
Whatever you fork it and maintain your own repo, or provide patches that can be easily applied. Dunno about IanB, but personally I don't even know how git works. I usually use SVN for my stuff which does everything that I need, so I don't want to learn git, nor clutter up my system with useless software. If things are the same for IanB (just with CVS in this case) I can understand this.

[IanB]

@Ultim,

Okay I made some time to look at your repository. From that I found 3 new very minor bugs. Very poor time invested per bug. I don't understand why you did not just reported them back in post #189 on 1st September 2013 instead of complaining about my going all red on you. Sorry you took the red text personally it was aimed at a much wider audience, it's a sad fact some simply cannot be bothered succinctly reporting the bug they find. Fortunately there are also many who do actually make the effort.

Yes I can see you have many ideas. Some I had already thought about for future versions, some new ones that I will take on board. Congratulations you are the first other person to recognise that we need _set_se_translator for MSVC versions later than 2K3.

It's a genuine pity that you could not resist the urge to move files around. You have a certain biased perspective (i.e it's Ultim's way only) that we did not share in so your restructuring just creates loss of reference without any personal benefit.

[ultim]

It's a genuine pity that you could not resist the urge to move files around. You have a certain biased perspective (i.e it's Ultim's way only) that we did not share in so your restructuring just creates loss of reference without any personal benefit.

*Sigh* You still don't give any reason why the files should not be moved, or why the places I moved them to are the wrong ones.

And please don't tell me that I only accept my way. I see it quite the contrary. I have written down literally half a dozen arguments why the structure matters and why the new structure is better, and even told you that I'll use any other structure as long as there is a reason for it. On the other hand, you just tell me "No, I won't move any files" without giving any justification. And of the two of us, I'm the biased?

But ok, I guess this is the most I will get. So I'll try to approach it from a different perspective. You told us yourself a couple of posts back that you've been itching to move file too, but you resist. Why do you resist? It is not like there are other code bases depending on your CVS, except for AvisynthMT.

So maybe after consulting with SEt, we could do it according to your dream-structure. Because it sure doesn't have to be my structure, I'd just like it to change, because the current structure sure is bad right now.

Maybe some background why I find it important to change: My goal is not just for me to improve Avisynth, but to allow it for more people from the community to join the development too. Currently it is very hard for anyone to do so, because of multiple reasons, like: unorganized files, the outdated development environment (VS6), unreadable code, missing commenting, the huge amount of exceptions thrown, and the choice of CVS is not the most contributor-friendly either.

This is why my current changeset focuses on improving developer friendliness, so that we lower the barrier for entry for new contributors. I understand, that unfortunately this means changes for existing developers too, mostly for you. But if we manage to attract more developers over time, this will pay off. The project structure plays a large role here. When a newbie comes, and sees source .cpp file of the core being in the distrib folder, some parts of non-essential plugins in the core, others parts in completely unrelated directories, "include" directories containing source files etc., he'll be confused, and with every right!

So, IanB, can we maybe reorganize the files according to how you've been wanting to have them?

[qyot27]

Some basic things related to the git history that I find I do want to comment on now as actual feedback, related to the file structure and general practices with git (and this also applies to generating out patches with 'git format-patch' for submission upstream).

I do feel the new structure is an improvement, but it's the presentation that I think needs to be done a bit better so it can be discussed and a more solid case can be made for why the individual choices made were made.

The renames, moves, adds, and deletes should be split into many smaller and separate commits, and the commit messages should fully (or as fully as possible) explain what is happening in each one, such as where files are getting moved/renamed to, or why binary files are deleted, etc.

Things that should have their own dedicated commits, based on what I've observed about the new structure:
Expanding the Examples out of their .zip archive
Deleting binary files from the git repository/history
The basic rename of the 'src' directory to 'avs_core' (and nothing else); this is probably 90% of the difference in the first place
Adding the docs to the history and noting where they are stored
Adding the CMake build system (if the CMake files for the newly-split plugins is purely for making these into separate plugins, leave those CMake files out for now; if they actually do have relevance for basic compilation as traditionally done, leave them in)
Removing the old build system's files, and explaining why they are being removed
Minor file renames - and make sure the renames are listed in the commit message itself.
Reorganizing of files to more appropriate locations (such as putting all source files related to converting things in the 'convert' directory); this should be separated into as many distinct commits as there are distinct subtopics for reorganizing (in other words, in the convert example I just gave, that is one commit; if there's another one related to some other destination directory, do it in a second commit, and so on)
Making Shibatch/ImageSeq/SoundTouch their own plugins (and each one should have its own commit) - IMO it's okay if any renaming and moving is consolidated into here in one patch, just so long as the moving/renaming is immediately pertinent to the goal of making the plugins distinct, and the rationale is - once again - explained in the commit message
Having these split into distinct commits makes it easier to take it piece by piece, explain the rationale for each change, and get approval or critiques from the upstream devs on what should be done. Explaining things in the commit messages and keeping them small in subject matter also doesn't give off the impression of a dramatic change and it's easier to see the immediate connection between where a file was and where it was moved to.

Another point more immediately related to git itself, is that the formatting of the commit messages should have a single concise summary as its headline, and then a longer explanation further down after a double-space. The lines in the commit message should be wrapped at 80 characters (the headline should be <= 80 characters in length). This is to make it easier for users pulling up the history with git log to view it without it overflowing their Terminal's width buffer (since most default to 80 characters), or for any need to use git rebase -i - if the lines exceed the width buffer, then it has the potential to interfere with git rebase -i in a really bad way.

Also, somewhat of a minor point, but let's say you want to focus on the development of a single issue, like multithreading. The work on this should be in its own branch, separate from master. This keeps the master branch clean so that integration of newer changes from upstream and rebasing the branches against master is easier and keeps the general noise down (IMO, development branches should always be considered volatile and subject to rebasing, even if the git repository is public; but never, ever, ever rebase master).

So another list of general git management points I think would make this easier:
Checkout a new branch for the changes you've made. All 54 of them that are currently on the github repo should be in a separate branch, and then reset --hard master's HEAD back to where upstream leaves off
Use git rebase -i on the new branch to edit the commit messages to comply with the 80 char limit and separate headline/explanation format, without changing the contents of any of the actual commits.
Use git rebase -i again, but this time to split the file structure changes into many smaller, more topical commits.



Also, I think there should be a new thread to discuss the changes ultim is proposing, so that the alpha4 thread doesn't get dragged any more off-topic (especially now that alpha5's been released).

[ultim]

Currently, the reason everything is on the main branch in my repo, is because it is my repo. Meaning it is set up so that if anyone clones my master, he'll get the latest stable version of my changes. That makes the most sense to me. Why would anyone clone my repo if they want the official branch of Avisynth? So my stable work is in the master of my repo, while I'd be using branches for non-stable or work-in-progress of other features I'm working on, and for upstream. This is not a problem for IanB, because he can pull changesets/patches from any of the branches, his master does not need to be the same as my master branch. If I were in the official repo though, I'd obviously be working in a seperate/private branch, though I doubt IanB will let me do that.

Concerning your other idea of splitting up the large file-moving commit, I'm okay with that. As long as I know beforehand that IanB is not gonna reject it again, I'd redo my changeset and split it up further as you proposed. So if he lets me know that he's willing and ready to integrate/merge the file moves that way, count me in. Of course we can discuss where files should go before I start redoing everything, so that I won't have to do it a 3rd time.

[qyot27]

Currently, the reason everything is on the main branch in my repo, is because it is my repo. Meaning it is set up so that if anyone clones my master, he'll get the latest stable version of my changes. That makes the most sense to me. Why would anyone clone my repo if they want the official branch of Avisynth? So my stable work is in the master of my repo, while I'd be using branches for non-stable or work-in-progress of other features I'm working on, and for upstream. This is not a problem for IanB, because he can pull changesets/patches from any of the branches, his master does not need to be the same as my master branch. If I were in the official repo though, I'd obviously be working in a seperate/private branch, though I doubt IanB will let me do that.
That suggestion was mostly to make sure that there's no potential conflicts if you try to pull in changes from upstream - having all of it in master means that there's the potential for changes to be sandwiched between runs of cvsimport, or for conflicts in the tree to arise that might be more unexpected (for that matter, the stuff from cvsimport could go in a different branch like 'trunk' or something, with master the way it currently is; the same basic idea applies - letting whatever happens upstream have its own branch). Github does allow the ability to set alternate branches as main so that a user doing a clone gets the branch you want them to get (I had it set up that way for my FFMS2* repo for a long time, where the 'patches' branch was the one you'd get from a normal clone).

*which is now the ffms2-old repo; I re-forked my main one against the official repo after the move to Github

[ultim]

Very unfortunately, me and IanB seem to have reached an impasse. I have received constructive PMs though from forum readers, some relating to how I present my patches, some regarding communications, but all encouraging. I'll be continuing my slow-ish efforts at improving Avisynth, and because upstream acceptance does not seem realistic ATM, I'll be forking the project (well, technically speaking, I already have). I wanted to avoid this, because forking has many downsides. Downsides like even more fragmentation, a lessened likelyhood of both projects benefitting from each other's improvements, and a bitter aftertaste in the mouth on both sides, I guess. The more optimistic amongst you can however spot advantages too, like not having to be constrained by the ideals and rules of the father project. I'll post more information about goals and such in my own thread.

Anyway, I just wanted to give you an update, to prevent you from thinking that I'm gone. This will be my last post "contaminating" this thread, and I'll create a thread of my own the next time, which will be when I post my next release. I have already synched with 2.6 alpha5 (synching was not a big deal), and have a large chunk a new code in place which completely (like, from scratch) replaces the plugin handling and loading. Also, Groucho2004 was right after all when he found something was off with out-of-memory handling in my build, so we'll see what I can do about that. It has to do with me using set_seh_translator and /EHa, which IS the correct thing to do on all VS2003 and above, but AFAICT it causes unhandled std::bad_alloc exceptions not to be swallowed silently anymore... which the project seemingly has relied upon? I've still got to take an exact look at that, but I know I'm on the right track. Anyway, back to my plugin-handling code, it makes script loading percievably faster if you have a lot of plugins, and it is also much cleaner from a developer's perspective. Oh, and expect support for multiple plugin directories too.

To finish forking, I guess the only thing left to do is to find a name for the fork. Feel free to send me any suggestions, I am torn between using something "Avisynth"-like or not. First of all, it'd be natural to do it, because my fork is Avisynth in its core. Admittedly, I am still relying on a large existing code base, so it'd be misleading to try to present the fork as something completely different. On the other hand, it would also be aggressive and disrespectful from me to have a name like Avisynth 3-4-5-6..., Avisynth-X, -NG, or who knows what suffix. So let me know what you think or if you have any ideas. I'm very interested in Ian's oppinion too, so I'd be thankful if he shared his preferences on naming with us. Of course I'm not asking him to come up with specific names for me, I'd just like to know what kinds of names are okay for me to use from an official perspective.

Cheers

[Groucho2004]

Also, Groucho2004 was right after all when he found something was off with out-of-memory handling in my build, so we'll see what I can do about that. It has to do with me using set_seh_translator and /EHa, which IS the correct thing to do on all VS2003 and above, but AFAICT it causes unhandled std::bad_alloc exceptions not to be swallowed silently anymore... which the project seemingly has relied upon?
Interestingly, SEt's MT DLL (built with VC2010) exhibits the same exception handling behaviour as the official one that is built with VC6.
No idea what he did, but I think he mentioned something about "hacking" the compiler.

[ultim]

Interestingly, SEt's MT DLL (built with VC2010) exhibits the same exception handling behaviour as the official one that is built with VC6.
No idea what he did, but I think he mentioned something about "hacking" the compiler.
That's because he is using different compiler settings. If I switch to /EHsc (which is the theoretical equivalent of VC6's /GX), I get similar exception behavior as in the official build. Of course then set_seh_translator falls apart. The official project solved set_seh_translator's absence using custom ASM-hacks, and I guess those are the hacks that SEt referred to. I got rid of those hacks in my build, as I prefer to use methods that are officially supported by the compiler. But now I need to fix up parts of the code to properly support this scenario.

On the other hand, it would also make sense to completely remove handling SEH-exceptions, because there are very few valid reasons to handle them, and even less that are worthy of showing to the user, as they indicate coding errors in Avisynth or in the plugins. I might really remove them in a later build of mine, but not now, to take it step-by-step.

[Groucho2004]

That's because he is using different compiler settings. If I switch to /EHsc (which is the theoretical equivalent of VC6's /GX), I get similar exception behavior as in the official build.
Not true for Windows XP. I tried that with VC7.1, VC9 and VC10, I always get this silly runtime message box instead of the actual message from "ThrowError()". I think it has something to do with msvcrt.dll on XP, not sure.

It would be nice if Avisynth could be built with newer compilers since there certainly have been improvements since VC6 but on the other hand, I have not come across any problems for which VC6 could be blamed exclusively.

[SEt]

Let me add some thought to the discussion.

Avisynth in current 2.6 form is effectively dead. I.e. it will work ok, but isn't endangered by any major improvements. There are several reasons for that:

1) The unsolvable one that makes it really dead and not just in bad shape or unmaintained: license. GPLv2 was bad choice – it should've been LGPL. I don't see any realistic way to change it to LGPL, so for that change all existing code needs to be thrown out and rewritten.
2) Bad interfaces. No one sane uses С++ for open plugin system. C interface with trivial wrappers to C++ or whatever else is what should've been done. In 2.6 we break old interfaces but get the same C++ hell.
3) No care for threading. Can you guess how many people need what was added in recent years (!) to official 2.6 and how many need working threading?
4) No care for submitted patches. Besides MT my build differs by several important source clarifications (specified calling conventions where they are needed, improved exception handling). How many years have they been open? Were they merged? (Note: most time I spend on updating my build to official is manually re-merging these small but extensive changes since CVS is often unable to automerge them.)
5) Absolutely refusing to remove old hacks, clean code and move to newer software. Who uses CVS today? With words VC6 most modern developers jump in panic or start to cry. How about this nice quote from sources:
// Warning! : If you modify this routine, check the generated assembler to make sure
// the stupid compiler is saving the ebx register in the entry prologue.
// And don't just add an extra push/pop ebx pair around the code, try to
// convince the compiler to do the right thing, it's not hard, usually a
// slight shuffle or a well placed "__asm mov ebx,ebx" does the trick.
And that is not a single case. The actual problem here is that, for example, SVN and VS2010 aren't any worse in general, much less buggy (no need for those register saving hacks) and actually used by people today. But do we need new developers? Not as much as we need CVS and VC6 obviously.


But saying that all I have to add that while Avisynth has plenty of problems there isn't anything better. And it works quite ok too.
I thought many times of rewriting it but still don't have enough resolve (due to 1 it's quite a daunting task) and it does what I need from it reasonably well (I would be pressed to write better threading if i had like 16-thread CPU, but no hardware upgrade here in 3 years and future doesn't look bright at all – looks like all focus of manufactures is shifted towards mobile).

[Mystery Keeper]

You're wrong about hardware, SEt. I'm currently aiming to get myself new PC with double 12-core Xeon. Which makes it 48 threads. As for "nothing better" - VapourSynth looks promising. You could join the development. Manao, Fizick and Tritical certainly should, so we could have motion compensation and high quality deinterlacing and denoising in VS.

[ultim]

Nice to see you SEt! Some of the problems you have listed are exactly why I started my fork. Obviously, I can't correct all of them, like the GPL issue. But we can help the rest! I agree that there isn't anything better yet than Avisynth, and while VapourSynth looks promising, Avisynth is still better in many aspects ATM, which is why I'm putting my energy into Avisynth.

CVS, VC6, the hacks, the C++ interfaces and all those /are/ bad, but these can be helped. I'm removing hacks like the one you listed one by one (many of them are already gone in my sources), and AFAICT porting to new software is finished - with the exception of a last bug that Groucho reported, but I've found the reason for that too. My hope is that by porting to newer software, getting rid of the old and deprecated cruft, and by being more open to submitted patches, we'll be able to breath some new life into Avisynth and invite new development. Additionally I'll be making small improvements here and there too. I hope to push my next set of changes to GitHub soon.

Groucho2004:
After some more investigation, I believe I have found the real reason for the changed exception behavior: http://stackoverflow.com/questions/550451/will-new-return-null-in-any-case .

[qyot27]

RE: naming, since all the other forks are in the same boat and just tack on a suffix (MT or 64 for example, although those are really just descriptive titles), I'd think it safe to just go ahead and do the same thing. As previously noted, I like 'AviSynth+', since it is a more general-purpose fork.

Also, the developer(s?) that worked on the dead 3.0 branch seem to be different than 2.x, and 3.0 changed things around too, yet it still kept the AviSynth name, so I wouldn't think that to be too aggressive either (especially if you pair it with a new fork name). I wasn't around for the early history, but if it seems to be a tendency for each of the major versions to have different groups of contributors working on them then there is a historical precedent involved.

How about collecting whatever suggestions there are for names and then putting it to a forum poll?

[vcmohan]

Request all who fork from the official version to please start separate threads and do not use the main thread. I request the moderators to remove already posted fork related matter to different threads

[turbojet]

I agree this talk should be moved into another thread once ultim comes up with a name and makes a thread but now that alpha 5 is out this thread would become inactive anyways.

To continue OT, I look forward to a fork with proper 64 bit handling and improved threading (if possible). Many FOSS go through forks and it gives ultim control to do what they want to do as opposed to being approved by IanB, which is a current issue. As far as name goes, isn't 'avi' a bit old now, would 'AVSynth' work?

[Groucho2004]

After some more investigation, I believe I have found the real reason for the changed exception behavior
Yes, simply linking with "nothrownew.obj" (which ships with VC10) keeps "new" from throwing "bad_alloc".

[Wilbert]

I moved all the relevant posts to this new thread. Let me know if i need to change the title.

[Mystery Keeper]

Not true for Windows XP. I tried that with VC7.1, VC9 and VC10, I always get this silly runtime message box instead of the actual message from "ThrowError()". I think it has something to do with msvcrt.dll on XP, not sure.

It would be nice if Avisynth could be built with newer compilers since there certainly have been improvements since VC6 but on the other hand, I have not come across any problems for which VC6 could be blamed exclusively.

The worst crime of VC6 is not following the standard. For example, a variable you declared as "for" loop counter continues to live outside the loop and you can't declare it again. That's only an example. I don't know how bad is its deviation. But I couldn't build AviSynth plugin with GCC. Meaning AviSynth headers code doesn't follow the standard either.

[qyot27]

But I couldn't build AviSynth plugin with GCC. Meaning AviSynth headers code doesn't follow the standard either.
Clarification: plugins using the [default] C++ interface can't be compiled with GCC. Part of the raison d'être for the C interface is to enable GCC to build plugins - the C interface itself also forms the backbone of the AviSynth input support in x264 (since late 2009) and FFmpeg (since March of this year).

But there's only a few plugins currently that actually make use of the C interface (off the top of my head, FFMS2's C plugin variant, AssRender, and yadif), and it's still subject to the huge caveat that C plugins aren't included in the autoloading feature.

As far as name goes, isn't 'avi' a bit old now, would 'AVSynth' work?
'AVSynth' seems like an FFmpeg function name conflict waiting to happen; if the desire is for a more general name, 'MediaSynth' would be more unique and less likely to conflict name-wise (but personally I'd view that as a last resort since the scripting language is still basically going to be AviSynth's, which is another reason why I think suffixing would be fine; other languages usually take a descendant diverging from them dramatically to invoke a name change, and often they use suffixing rather than completely changing the name, look at all the different C family members, all of which probably diverge from C more than ultim's changes will end up diverging from current AviSynth's scripting language).

IMO, 'avi' being old has little bearing on the name. When using NLE software, lossless codecs are still usually best transported in AVI (if you aren't using something Apple-specific or device-specific, anyway) due to the least common denominator familiarity with that container and the simple one frame in, one frame out I/O mechanism of Video for Windows, and the basic I/O features of an AviSynth script still match the way AVIs are used with lossless codecs (even with projects moving away from accessing AviSynth through Video for Windows and just accessing the library directly).

[turbojet]

All I could find for avsynth is https://github.com/LRN/gst-avsynth which seems abandoned, does ffmpeg have something in the pipe named AVsynth?

I agree avi is still useful but modern lossy encoders have given it a bad name and often seen as a last resort or unsupported.

[ultim]

... and it's still subject to the huge caveat that C plugins aren't included in the autoloading feature.

Since when? Oh sorry, I've forgot you're not using ultim's build ;)
I wanted to get a release out yesterday in the evening (CET), but I've realized last minute that the new shadow plugin folder feature's API could be improved, so I've delayed. I hope I'll be able to release tonight, but no guarantees, since I'll be getting home later today.

And yes, the above means C plugins are autoloaded with my rewritten plugin management, I worked on that since I was also missing that feature, and find it very useful and even necessary to encourage using it.

Yes, simply linking with "nothrownew.obj" (which ships with VC10) keeps "new" from throwing "bad_alloc".
That works too, but I prefer a standards compliant implementation so I've chosen another way. I went ahead and inspected where "new" is used to find places where we depended upon it returning NULL, and changed those occurrences to the non-throwing (std::nothrow) version. This way we are not only compliant to the c++ standards, but we also don't require to link with special MS-specific object files.

p.s. I still haven't decided on a name so keep 'em coming :)

[qyot27]

All I could find for avsynth is https://github.com/LRN/gst-avsynth which seems abandoned, does ffmpeg have something in the pipe named AVsynth?
That's not what I was saying. Naming conventions for lots of API and function stuff in the libav* libraries start with AV_ - so steering clear of stuff approaching that would be advisable.

Since when? Oh sorry, I've forgot you're not using ultim's build ;)
I wanted to get a release out yesterday in the evening (CET), but I've realized last minute that the new shadow plugin folder feature's API could be improved, so I've delayed. I hope I'll be able to release tonight, but no guarantees, since I'll be getting home later today.

And yes, the above means C plugins are autoloaded with my rewritten plugin management, I worked on that since I was also missing that feature, and find it very useful and even necessary to encourage using it.
I was hoping as much when plugin management got mentioned before. Huzzah.

I've been refining the build instructions some, now that I've also got the DirectShowSource-related stuff documented step-by-step.

[raffriff42]

Just brainstorming here...
MediaSynth
Avisinthe
ASST (AviSynthyServerThingy)
AINRA (AINRA Is Not Really Avisynth)
AVI Normal (does this need explaining? Young Frankenstein?)

[kypec]

ModAviSynth is my proposal which should define this fork as modern and modified project at the same time. Thumbs up, ultim and :thanks: for all your hard work on improving AviSynth.
VapourSynth looks promising but not yet quite there. ;)

[ultim]

Thank you everybody for the suggestions. Right now I'm leaning towards qyot27's "AviSynth+". I find it good because it expresses exactly what the project is, and what its intention is. It makes it easy to distinguish between the two projects, while it still notes that it is rooted in Avisynth, and has a clear message that it is supposed to be an enhanced version of the original project.

[mastrboy]

Wouldn't a single "+" make it a little hard to search for? Considering it's interpreted as an operator by most search engines?

[qyot27]

The + could simply be the stylized name that appears in say, the output of Version(). Having it written out as 'Plus' in other places like the Github repo URL would be fine.

[Reel.Deel]

+1 for "AviSynth+", I've tried to think of other names but that's a tough one to beat.

[raffriff42]

Buy the domain name now, or it might get squatted upon. ;)

[hanfrunz]

What avisynth development really needs is a meeting of some of the core developers in person. I have no idea if this is possible, but an avisynth developer conference/camp would be cool. Maybe you find another conference, where you could meet. For example the 30c3 (https://events.ccc.de/congress/2013/wiki/Main_Page) in Germany this december, etc.

[Groucho2004]

What avisynth development really needs is a meeting of some of the core developers in person.
As far as I know, IanB is The Last Man Standing from that group.

[ultim]

Yuhuuuu, new build, and new code on GitHub!

So, what has changed:
- First of all, the crash-on-out-of-memory bug is hopefully fixed. Should be.
- There is a brand new plugin-system in place, and if you work with scripts that use a lot of plugins, you should notice that they load faster.
- You can have multiple plugin directories. Exact semantics in my next post.
- LoadCPlugin (or Load_Stdcall_Plugin) is now a synonym for LoadPlugin. LoadPlugin will load C-plugins and LoadCPlugin will load normal plugins. They are one and the same. No difference.
- Hence, C plugins are also autoloaded.
- LoadVFAPIPlugin() is out of order for now. I'm not planning on removing it, I just need some info how to correct it.

Changes noteworthy for developers:
- Invoke finally stops throwing exceptions as a "normal condition" -> better debuggability
- VFAPI and VirtualDub filter loading are now separated into their own plugins, and are not in core any more.

[ultim]

Okay, so how do multiple plugin directories interact with plugin autoloading?

As a recap, here is how it used to work in the official Avisynth:
- Look for the string HKEY_CURRENT_USER/Software/Avisynth/PluginDir2_5 in the registry. If it exists, load plugins from the path specified there and stop.
- If the above string didn't exist, look in HKEY_LOCAL_MACHINE/Software/Avisynth/PluginDir2_5. Try to load plugins from the path specified there.
- Done.

First thing to note, is that classic Avisynth only ever searches for plugins in one single directory. It only knows two directories (both specified in the registry), and it only tries the second path if there is no entry for the first one.

Avisynth+'s autoloader has a list of autoload directories. It iterates over all those directories and tries to load all plugins from each. But (and a big but!) it will not load a plugin from a directory if another plugin with the same basename is already loaded. The basename of a plugin is simply its file name without the extension.

The expected use case is that you can now overlay a new plugin directory on top of another one. Avisynth+ then would load all plugins from the first folder, then load only those plugins from the second that weren't loaded from the first, then those from the third that weren't loaded from the first or second and so on. For example, let's say your usual plugin folder has a lot of plugins you normally use. But at one time you have a small number of updated plugins that you only want to use from a few scripts, but you do not yet want to replace your existing plugins globally. Then you'd just add a new plugin overlay folder, with only the new plugins in it, and that's it. All scripts that specify the new folder will autoload all plugins from your usual one, except for the new plugins, which would get loaded from the new folder. All your other scripts will still use your old plugins.

My next post will tell you how to add and remove autoload folders.

[Groucho2004]

Avisynth+'s autoloader has a list of autoload directories. It iterates over all those directories and tries to load all plugins from each. But (and a big but!) it will not load a plugin from a directory if another plugin with the same basename is already loaded. The basename of a plugin is simply its file name without the extension.
Might be a bad idea. Take for example the RemoveGrain package. It has several DLLs, each built with different optimizations (SSE, SSE2, etc.). They are named "RemoveGrainSSE.dll", "RemoveGrainSSE2.dll", you get the idea. Yet, they all have the same name in AvisynthPluginInit. I bet that 3 out of 4 users dump the lot in their autoload directory.

[Groucho2004]

First of all, the crash-on-out-of-memory bug is hopefully fixed. Should be.
Yes, it is. :)

[ultim]

By default, Avisynth+'s autoload folder list has four paths in it, in this order:
- PluginDir+ in Software/Avisynth in HKEY_CURRENT_USER
- PluginDir+ in Software/Avisynth in HKEY_LOCAL_MACHINE
- PluginDir2_5 in Software/Avisynth in HKEY_CURRENT_USER
- PluginDir2_5 in Software/Avisynth in HKEY_LOCAL_MACHINE

This means, if there are ever plugins which will only work with Avs+ but not with classic Avs, you can put them into one of the "PluginDir+" folders. Avisynth+ will then use the classic plugins from the normal Avisynth, but if there are versions of some plugins written for Avisynth+, it will use them instead, and the classic avisynth.dll will still not be bothered with them. This is all without you having to lift a finger (except for adding the "PluginDir+" values to the registry once, until we have an installer). So to summarize all this, you have the ability to define a plugin autoload folder in the registry which will only be used by Avs+, but not by Avs, in addition to your classic plugins.

However, another new functionality offered by Avisynth+, is that now you can also specify autoload paths in the scripts. There are two functions for this:

- AddAutoloadDir(string path, bool toFront): this will add a new autoload folder. The string parameter is obligatory, it is the folder path where to load from. The second boolean parameter is optional, and if true (default), it will add the path to the front/beginning of the autoloader's list, which means it will be searched earlier than the rest. If it is false, the path will get added to the end of the list, so it will get searched last (unless you again add another one to the end).
- ClearAutoloadDirs(): This will clear all the paths from the autoloader's list. Note that it is NOT a reset to the default state. ClearAutoloadDirs() will clear all folders, so if you don't add new ones after that, you have disabled the autoload functionality. This is, BTW, also a way to disable autoloading for a particular script in Avisynth+.

Here's an important note: You can only call these functions if no plugin has been autoloaded yet. Autoloading happens if the first unknown function is looked up. This means you can only call AddAutoloadDir or ClearAutoloadDirs if you have only made calls to built-in functions up to that point in the script. I suggest you start your scripts with these calls to avoid any problems.

There is only one thing left to discuss: Are there any special directories you can reference from your script? You bet there are :)
- SCRIPTDIR is the folder of the most current script. It is the path of the imported script if your script calls import()
- MAINSCRIPTDIR is the folder of your main script, the one where execution started
- PROGRAMDIR is the folder of the executable running the current script
- USER_PLUS_PLUGINS is the string stored in PluginDir+ in Software/Avisynth in HKEY_CURRENT_USER
- MACHINE_PLUS_PLUGINS is the string stored in PluginDir+ in Software/Avisynth in HKEY_LOCAL_MACHINE
- USER_CLASSIC_PLUGINS is the string stored in PluginDir2_5 in Software/Avisynth in HKEY_CURRENT_USER
- MACHINE_CLASSIC_PLUGINS is the string stored in PluginDir2_5 in Software/Avisynth in HKEY_LOCAL_MACHINE
... all these special constants are case-sensitive for now.

Examples
- If you want plugins to be autoloaded from the script's "autoload" directory too, you'd write:
AddAutoloadDir("MAINSCRIPTDIR/autoload")

- If you want plugins to be autoloaded from the script's "autoload" directory, only from there and nowhere else, you'd write:
ClearAutoloadDirs()
AddAutoloadDir("MAINSCRIPTDIR/autoload")

- If you wanted to manually recreate the default state of the autoloading folder list, you'd write:
ClearAutoloadDirs()
AddAutoloadDir("USER_PLUS_PLUGINS", false)
AddAutoloadDir("MACHINE_PLUS_PLUGINS", false)
AddAutoloadDir("USER_CLASSIC_PLUGINS", false)
AddAutoloadDir("MACHINE_CLASSIC_PLUGINS", false)

[ultim]

Might be a bad idea. Take for example the RemoveGrain package. It has several DLLs, each built with different optimizations (SSE, SSE2, etc.). They are named "RemoveGrainSSE.dll", "RemoveGrainSSE2.dll", you get the idea. Yet, they all have the same name in AvisynthPluginInit. I bet that 3 out of 4 users dump the lot in their autoload directory.

Yes, in this case they'd be "recognized" as separate plugins. But even if Avs+ could tell that they are different versions of the same plugin, it still wouldn't be able to decide which one to load (for that it'd neeed some mechanism to determine the used optimizations in a binary, which is not really feasable). So the choice of a different basename wouldn't really help here anyway, AFAICT. The problem is really that RemoveGrain doesn't use dynamic dispatching of routines based on the current machine's instruction set.

[Groucho2004]

The problem is really that RemoveGrain doesn't use dynamic dispatching of routines based on the current machine's instruction set.
Removegrain was just an example and yes, the missing CPU dispatcher is the problem.
Another thing is to improve somehow the detection of garbage in the autoload directories (i.e. old plugins which are incompatible with 2.6). However, querying the interface version for each plugin every time would probably be too time consuming.

[qyot27]

What about an index that caches the necessary functions out of the plugins (maybe something like what fontconfig does, where the cache is only regenerated when it detects a change in the directory contents)? Would that allow more flexibility to determine which version of a plugin should be used or which plugins actually get autoloaded? Checking a cache file would probably be a lot faster than checking the plugins themselves, right?

[Gavino]

- If you want plugins to be autoloaded from the script's "autoload" directory too, you'd write:
AddAutoloadDir("MAINSCRIPTDIR/autoload")
In what contexts do MAINSCRIPTDIR and the other 'special' names get replaced with the corresponding folders?
In all strings, or only when used in the argument to AddAutoloadDir?

[qyot27]

On the C-plugin autoloading front, I can verify that AssRender now autoloads, but FFMS2's C-plugin (github.com/FFMS/ffms2/commits/c_plugin) does not, even when I try to load it with LoadPlugin.

On a guess, I think it *might* have to do with FFMS2 using the same modified-for-2.6-colorspaces version of avisynth_c.h that x264 and FFmpeg do. So a feature request would be to merge in those changes and finally bring avisynth_c.h up to speed.

[qyot27]

The build instructions I posted earlier have been revamped and fleshed out. It now includes DirectShowSource setup.

[ultim]

Another thing is to improve somehow the detection of garbage in the autoload directories (i.e. old plugins which are incompatible with 2.6). However, querying the interface version for each plugin every time would probably be too time consuming.
I'm not sure if that is what you mean, but both Avs and Avs+ already query interface versions. They try to load the 2.6 interface from a plugin first, and if that is not supported, they try to load the 2.5 interface. Avs+ also tries to load the C interface if both of the previous ones fail. In the future, the C interface should probably be prioritized over 2.5.

What about an index that caches the necessary functions out of the plugins (maybe something like what fontconfig does, where the cache is only regenerated when it detects a change in the directory contents)? Would that allow more flexibility to determine which version of a plugin should be used or which plugins actually get autoloaded? Checking a cache file would probably be a lot faster than checking the plugins themselves, right?
I'm not sure how that would help. Even if it detected a plugin was changed/added/removed, that still doesn't tell us much about whether it is the "correct" version. Can you elaborate on your idea further?

As a side note, because Avs(+) doesn't have a continuously running process in the background, plugin change detection cannot be implemented efficiently. Even fontconfig has to be notified from an external application if the font folder changes. And purely for performance, implementing a plugin cache isn't needed for Avisynth. As a comparison, there are much fewer plugins than fonts, and the plugins are also much smaller.

In what contexts do MAINSCRIPTDIR and the other 'special' names get replaced with the corresponding folders?
In all strings, or only when used in the argument to AddAutoloadDir?
Only in AddAutoloadDir(), and even there, only if they are at the very beginning of the string. These get replaced to absolute folder paths, so if they are not at the beginning of the string, replacing them would only result in an invalid path (e.g. you'd end up with "c:" in the middle of your path).

On the C-plugin autoloading front, I can verify that AssRender now autoloads, but FFMS2's C-plugin (github.com/FFMS/ffms2/commits/c_plugin) does not, even when I try to load it with LoadPlugin.

On a guess, I think it *might* have to do with FFMS2 using the same modified-for-2.6-colorspaces version of avisynth_c.h that x264 and FFmpeg do. So a feature request would be to merge in those changes and finally bring avisynth_c.h up to speed.
Nice spotting. I'll have a look at it when I get home.

The build instructions I posted earlier have been revamped and fleshed out. It now includes DirectShowSource setup.
Thx.

[qyot27]

I'm not sure how that would help. Even if it detected a plugin was changed/added/removed, that still doesn't tell us much about whether it is the "correct" version. Can you elaborate on your idea further?

As a side note, because Avs(+) doesn't have a continuously running process in the background, plugin change detection cannot be implemented efficiently. Even fontconfig has to be notified from an external application if the font folder changes. And purely for performance, implementing a plugin cache isn't needed for Avisynth. As a comparison, there are much fewer plugins than fonts, and the plugins are also much smaller.
The original idea actually was rooted in how I understood the original AviSynth's plugin autoloading to work, where all plugins in the autoload folder were loaded so it could identify where functions were, then everything was unloaded, and finally the necessary ones were loaded again. That initial step could cause issues if a plugin was prone to crashing, even if said plugin wasn't even used in the script (one of the variants of WarpSharp was known to cause issues like this).

The cache idea was mostly thought up to avoid that scenario, since it would store the names and plugin locations of the functions and avoid having to perform those first two steps and any instability that could cause - instead, it'd read the cache, compare it to the script, and then load only the plugins used in the script.

But if there's an issue with plugin name collisions, a cache might allow for user overrides on a plugin-specific level while still keeping the new directory priorities the same (in the original project, I'm pretty sure this is why the ability to prefix the function with the name of the dll was added: dgdecode_MPEG2Source vs. mpeg2dec3_MPEG2Source, even though that's the case of the same function, but different .dll names - having it user-customizable would remove the need for special prefixing since the user could set whichever plugin they want as the default, and it would cover cases if the plugin and function names were the same, since the user could have the ability to set an alias of their own in the cache). Or in the RemoveGrain example, it could use a separate CPU dispatching plugin or function and the cache could be used to prioritize which version would take priority on the host CPU.



Regarding FFMS2, I don't think the extra 2.6 stuff in its version of avisynth_c.h was preventing it from autoloading, since I experimented with it a little last night using Meld to merge the changes in locally, and it still didn't autoload. So it seems the problem lies elsewhere (although having avisynth_c.h get updated would be nice).

[ultim]

...
The cache idea was mostly thought up to avoid that scenario, since it would store the names and plugin locations of the functions and avoid having to perform those first two steps and any instability that could cause - instead, it'd read the cache, compare it to the script, and then load only the plugins used in the script.
...
Ah, now I understand. So you mean a separate cache instance per script. Yes, that should work, and makes sense as long as the cache format is chosen so that it is easily editable by the user. The problem I see though, is that the cache would need to be regenerated each time the contents of plugin folders are changed, including non-plugin files. Even if the script has not changed. This equals to rescanning all the plugins, we don't have to execute code from all of them though, only in new or changed files.

Regarding FFMS2, I don't think the extra 2.6 stuff in its version of avisynth_c.h was preventing it from autoloading, since I experimented with it a little last night using Meld to merge the changes in locally, and it still didn't autoload. So it seems the problem lies elsewhere (although having avisynth_c.h get updated would be nice).
I didn't have time yesterday again, so all I could manage was to verify that the precompiled 2.17 Cplugin works (as a Cplugin, ofc). It works. Today I'm gonna recompile the cplugin from latest git and see what's the situation with that.

[qyot27]

I didn't have time yesterday again, so all I could manage was to verify that the precompiled 2.17 Cplugin works (as a Cplugin, ofc). It works. Today I'm gonna recompile the cplugin from latest git and see what's the situation with that.
Could it have to do with the dual handling of 2.5 and 2.6 (even though that's present in 2.17 also)? The C-plugin explicitly has a 2.5 mode and a 2.6 mode. You can see this in action by throwing FFmpeg into gdb and using a script using FFMS2 with either 2.5.8 or 2.6 - the language changes in the debug messages.

With the latest 2.6 MT (the video file being served in was 10-bit 4:4:4 H.264):
(gdb) r -i test.avs
...
libpostproc 52. 3.100 / 52. 3.100
Error while mapping shared library sections:
`C:\WINDOWS\system32\avisynth.dll': not in executable format: File format not recognized
warning: FFMS2 avs plugin: Initializing...
warning: FFMS2 - avs 2.6 mode
Guessed Channel Layout for Input Stream #0.1 : stereo
Input #0, avisynth, from 'test.avs':
Duration: 00:05:00.47, start: 0.000000, bitrate: 0 kb/s
Stream #0:0: Video: rawvideo (444P / 0x50343434), yuv444p, 848x480, 23.98 fps, 23.98 t
br, 23.98 tbn, 23.98 tbc
...

vs. 2.5.8:
(gdb) r -i test.avs
...
libpostproc 52. 3.100 / 52. 3.100
warning: DllMain: hModule=0x10000000, ulReason=1, lpReserved=0x00000000, gRefCnt = 0

warning: FFMS2 avs plugin: Initializing...
warning: FFMS2 - avs 2.5 mode
Guessed Channel Layout for Input Stream #0.1 : stereo
Input #0, avisynth, from 'test.avs':
Duration: 00:05:00.47, start: 0.000000, bitrate: 0 kb/s
Stream #0:0: Video: rawvideo (BGRA / 0x41524742), bgra, 848x480, 23.98 fps, 23.98 tbr,
23.98 tbn, 23.98 tbc
...


I'm fairly sure I'm still using the same build I posted here, if it helps:
http://forum.doom9.org/showthread.php?p=1643738#post1643738

But building right from git would work, too. All the changes to the c_plugin branch after that build were for Linux/OSX (and the currently-disabled HEVC tag).

[ultim]

I can't quite remember *when* 2.17 was tagged, but I'm thinking it probably was from right before it gained the dual 2.5/2.6 handling. The C-plugin explicitly has a 2.5 mode and a 2.6 mode. You can see this in action by throwing FFmpeg into gdb and using a script using FFMS2 with either 2.5.8 or 2.6 - the language changes in the debug messages.
Yes I noticed that some builds have dual interfaces, in my case C and 2.5. I made sure that the C interface of ffms2 was used by temporarily changing the interface load order in Avs+, and ffms2 loaded up and worked correctly with the C interface. That was 2.17 though.

I'm fairly sure I'm still using the same build I posted here, if it helps:
http://forum.doom9.org/showthread.php?p=1643738#post1643738

But building right from git would work, too. All the changes to the c_plugin branch after that build were for Linux/OSX (and the currently-disabled HEVC tag).
Oh, thx for the link. I'll just use that binary then instead of building it myself :)

[qyot27]

Well, unfortunately, I think it's actually a problem with my computer. I went and tried the official 2.17-cplugin package, and it won't load either.

[avisynth @ 03792520] Script error: There is no function named 'ffindex'.
(C:/Program Files/AviSynth 2.5/plugins/FFMS2.avsi, line 32)
(test.avs, line 1)
test.avs: Unknown error occurred
or if I don't try to use FFMS2.avsi:
[avisynth @ 03792520] Script error: There is no function named 'FFVideoSource'.
(test.avs, line 1)
test.avs: Unknown error occurred

So something is going on that makes it fail, but I don't know what.

[ultim]

I tried your build from http://forum.doom9.org/showthread.php?p=1643738#post1643738 , and it worked for me. If the problem on your computer is a missing dependency, depends.exe can help you diagnose what is wrong.

[qyot27]

I don't think it's a missing dependency, because builds of the previous HEAD (BitBlt refactor) can load it, and looking at that build and the current HEAD in Dependency Walker show that the only three missing ones - EFSADU.DLL, IESHIMS.DLL, and WER.DLL - are missing from both of them. Did rewriting the plugin management bring in a new dependency?

I'm going to try running a Debug build and see if it catches anything.

[qyot27]

I really can't figure out the MSVC debugger, but both DebugView and gdb picked up on the following when I tried to run a script with a Debug build of AviSynth+ in ffmpeg:
warning: DllMain: hModule=0x10000000, ulReason=1, lpReserved=0x00000000, gRefCnt = 0

warning: StringDump: Allocating new stringblock.

[avisynth @ 03792c00] Script error: There is no function named 'ffindex'.
(C:/Program Files/AviSynth 2.5/plugins/FFMS2.avsi, line 31)
(test.avs, line 1)
warning: StringDump: DeAllocating all stringblocks.

test.avs: Unknown error occurred
warning: DllMain: hModule=0x10000000, ulReason=0, lpReserved=0x00000000, gRefCnt = 0
Dunno if that helps.



I did try to bisect it, though, and the issue arises with the rewritten plugin management in 9567b94e. The commit immediately before it works. I'm going to see if it's okay on some of the other computers I have access to.

[ultim]

Thx for your efforts. I'm debugging it right now too. I've narrowed down the problem onto the 2.5 interface not working with ffms2, but dunno yet why. Don't try it on other machines, it won't work either.

The reason it worked for me, is because to make sure that ffms2 will be loaded with the C interface (and not 2.5), I've changed the interface probation order on my computer. Loading ffms2's does really work then, and I'm 100% sure it does on your computer too.

The problem is, that the binary I've published searches for the Avs 2.5 interface before the C interface, and because ffms2 also has that, Avs+ will use that instead of the C interface. This is where things start to get interesting. The call is made to the AvisynthPluginInit2 function, but it never registers any functions. I see it in your sources that it should correctly call Env->AddFunction, yet during runtime those calls never get back to me. I still have to find out why.

[ultim]

The problem seems to be in your build of ffms2. If I use TheRyuu's binary of 2.19 (the post right after yours in http://forum.doom9.org/showthread.php?p=1643738#post1643738), then it works perfectly. If I then overwrite ffms2.dll with your build r836, it stops working again.

Whatever the differences are in the source or compilation steps between your and TheRyuu's binary, it breaks your Avisynth 2.5 interface in ffms2.

[qyot27]

The problem seems to be in your build of ffms2. If I use TheRyuu's binary of 2.19 (the post right after yours in http://forum.doom9.org/showthread.php?p=1643738#post1643738), then it works perfectly. If I then overwrite ffms2.dll with your build r836, it stops working again.

Whatever the differences are in the source or compilation steps between your and TheRyuu's binary, it breaks your Avisynth 2.5 interface in ffms2.
The difference between them is that TheRyuu's build is the regular C++ plugin from the master branch, not the C plugin. The C plugin sources are in the src/avisynth_c directory (which only exists in the c_plugin branch), and completely ignores the src/avisynth directory since that's the C++ interface.

[qyot27]

And after experimenting with the checking order in avs_core/core/PluginManager.cpp, swapping the C and 2.5 checks, FFMS2 does indeed work. But then AssRender stops loading.

EDIT: Totally disregard the AssRender thing, I'd forgotten I'd moved it out of the plugins folder.

[ultim]

The difference between them is that TheRyuu's build is the regular C++ plugin from the master branch, not the C plugin. The C plugin sources are in the src/avisynth_c directory (which only exists in the c_plugin branch), and completely ignores the src/avisynth directory since that's the C++ interface.

Even if you say that the Cplugin-build completely ignores the other directory, an entry point for AvisynthPluginInit2 is still generated in the Cplugin, which makes Avs(+) think that it supports the C++ interface. I'd suggest removing that method (or at least not exporting it) in the C plugin build if it is officially not working anyway, to prevent Avisynth servers from trying to load that interface. Then everything will work.

Whether you remove the C++ entry point from the C build or not, I'm changing the loading order in my next public build. Meaning that the new order will be 2.6 -> C -> 2.5. Not just because of ffms2, I've already written in #74 (http://forum.doom9.org/showpost.php?p=1646392&postcount=74) (that was before you've mentioned your problem with ffms2) that this change should happen. ffms2 is just yet another reason. But I still think you should remove AvisynthPluginInit2 from the C-build if it is not intended to work anyway.

[qyot27]

It's true that the C plugin doesn't build anything from the src/avisynth directory, in that the GNUmakefile never refers to it. The reference to AvisynthPluginInit2 is in src/avisynth_c/avisynth.c, though, which is where all of this ends up getting caught.

The offending code block in question:
/* the AVS loader for LoadPlugin.
* Allows the conditional avs script logic:
* ( LoadPlugin("ffms2.dll") == "Use LoadCPlugin" ) ? LoadCPlugin("ffms2.dll") : NOP()
* to successfully load the plugin for both MSVC and MinGW versions */
const char * __stdcall AvisynthPluginInit2( void *Env )
{ return "Use LoadCPlugin"; }

So it would appear it was put in there for some sort of LoadPlugin trickery, but I can't see any reason for it either. I've removed it locally so that the C-plugin only uses the C interface init.

[ultim]

/* the AVS loader for LoadPlugin.
* Allows the conditional avs script logic:
* ( LoadPlugin("ffms2.dll") == "Use LoadCPlugin" ) ? LoadCPlugin("ffms2.dll") : NOP()
* to successfully load the plugin for both MSVC and MinGW versions */
const char * __stdcall AvisynthPluginInit2( void *Env )
{ return "Use LoadCPlugin"; }

Because Avisynth has been supporting both Cplugins and C++ plugins for a long time now, that piece of code doesn't look too meaningful for me either. Both MSVC and MinGW versions of ffms2 should be able to load using LoadCPlugin, so instead of writing that condition into a script, the script author could simply just use LoadCPlugin. I think that hack must be really old.

Anyway, with that removed, my previously posted binary build of Avs+ should load your ffms2 correctly now.

[Gavino]

/* the AVS loader for LoadPlugin.
* Allows the conditional avs script logic:
* ( LoadPlugin("ffms2.dll") == "Use LoadCPlugin" ) ? LoadCPlugin("ffms2.dll") : NOP()
* to successfully load the plugin for both MSVC and MinGW versions */
const char * __stdcall AvisynthPluginInit2( void *Env )
{ return "Use LoadCPlugin"; }

So it would appear it was put in there for some sort of LoadPlugin trickery, but I can't see any reason for it either.
It makes sense in vanilla Avisynth, as you can use the highlighted line in a script to load ffms2.dll regardless of whether you have the C++ or C version.

With ultim's build, that's not necessary (and actually goes wrong) since LoadPlugin and LoadCPlugin are the same thing.

[ultim]

It makes sense in vanilla Avisynth, as you can use the highlighted line in a script to load ffms2.dll regardless of whether you have the C++ or C version.

With ultim's build, that's not necessary (and actually goes wrong) since LoadPlugin and LoadCPlugin are the same thing.


AFAIK, vanilla Avisynth also has C-plugin support, so it doesn't seem to make sense to me, because instead of writing "( LoadPlugin("ffms2.dll") == "Use LoadCPlugin" ) ? LoadCPlugin("ffms2.dll") : NOP()" the script author could say "LoadCPlugin("ffms2.dll")" in both projects.

But more importantly, with these changes the plugin should now be able to load in both vanilla Avisynth and in Avisynth+.

[Gavino]

AFAIK, vanilla Avisynth also has C-plugin support, so it doesn't seem to make sense to me, because instead of writing "( LoadPlugin("ffms2.dll") == "Use LoadCPlugin" ) ? LoadCPlugin("ffms2.dll") : NOP()" the script author could say "LoadCPlugin("ffms2.dll")" in both projects.
But that won't work if the ffms2.dll you have is the C++ version. The point of using the conditional line in the script is that the user doesn't need to know which version he actually has if he loads it that way.

However, he could also use a try/catch instead and that would not require the extra code in the C plugin. The same script would then work in both Avisynth and Avisynth+. (In avs+, the try/catch would be unnecessary, but harmless).

[ultim]

But that won't work if the ffms2.dll you have is the C++ version. The point of using the conditional line in the script is that the user doesn't need to know which version he actually has if he loads it that way.
Ah, now I finally get it. I was assuming all the way through that the user would be using the Cplugin version of ffms2. Thx for making it clear to me.

As I said before, I'm gonna change Avisynth+ to try the C interface of a plugin before the 2.5 interface anyway. That will solve these problems too, and the user would be able to use both the C and C++ version of ffms2 with any of the two Avisynth projects, even if qyot27 decides not to remove those lines. I guess it is even better then if leaves those lines there, for the users who are using that hack.

[qyot27]

I actually did remove them, because the real need for that hack is only if ffms2.dll is not in the autoloading folder. The C-plugin (well, my builds of it) has long had LoadCPlugin("ffms2.dll") included in FFMS2.avsi, which handles it for classic AviSynth, and the C++ version would autoload on classic anyway. The inclusion of LoadCPlugin in FFMS2.avsi got merged into the main c_plugin branch after FFMS2 officially moved to git about a month ago.

I've not issued a pull request for it yet, so it's only in my personal FFMS2 repo.

[TurboPascal7]

Great job, ultim, I'm really happy to see someone not ancient working on avisynth.
What I would like to see is some kind of a roadmap. In a form of github issues or just a page in the wiki, but anything that would help developers start contributing would be really helpful. I assume x64 and multithreading support is planned in the (not near) future?

And two more questions:
1. What is the reason to keep compatibility with 2005 runtime when you can target XP with 2008/10/11 just fine (11 might be having some problems in wine though)?
2. What exactly breaks AvsPmod compatibility? This problem is really important since it renders the project useless for a lot of people.

[jpsdr]

I also be happy to see an x64 version, but from what i've understood, two big steps are needed.
1 - Remove all the inline asm and put it on external asm files.
2 - Rewrite these asm file to be x64 compliant.
The work already done in the already existant x64 version may help, nevertheless, it still seems to be a very big work...

[TurboPascal7]

I wouldn't mind helping rewriting the asm to intrinsics but I don't think I can handle external asm files for long (nor do I think it's reasonable but that's another question). That's the main reason I asked about vs2005 support - I'm not sure how good it is with intrinsics and using external asm might be the only option.

[qyot27]

I'd mentioned it in a PM, but AvxSynth (which can build and work on x64 just fine) did have asm disabled and replaced with regular C/C++ versions that could get re-used, although for all I know it may have just merged that stuff in from Avisynth64. From what I understand there was a push a couple years ago or so to convert the asm in AviSynth to x264ASM, which would result in separating the stuff out anyway. Not to mention cross-platform interests and the possibility of non-x86 CPU architectures.


On a different tangent, I'd prefer if there was a more project-specific setup on Github. It's a lot easier to remember, say, github.com/avisynthplus/avisynthplus.git than having it as simply a personal repo of avisynth as it currently is. For that matter, shouldn't the thread title also be changed now too?

[vdcrim]

2. What exactly breaks AvsPmod compatibility?
Just fixed in current git.

The $InternalFunctions$ and $PluginFunctions$ global vars are gone since the plugin management rewrite. Was that by mistake?

[ultim]

I will add issues to the Wiki to start with, and most probably I should also add a main README.md to the GitHub project so that visitors there get some more information other than the dump of a repository.

Right now I'm still looking at exception usage in Avisynth, continuously doing smaller cleanups and refactorings. I've also started to add Gavino's scripting extensions to Avisynth+, but I had to ask him because GScript and Avisynth have slightly different licensing terms. Fortunately he is completely okay with the integration (I've just received reply), so I can continue with that. I've also spent some time since the last release inspecting alternatives to the current BitBlt, so let us hope that it will result in a slight performance improvement. If there will be any, it won't be large, but at least it will be uniform (not specific to certain plugins or scripts).

Adding good threading is on the top of my wish-list too, and I am trying to find ways. As of right now there are parts of the Avisynth code involved which I did not investigate fully yet (mostly related to caching), so first I need to figure that part out completely.

x64 support is also a goal, but certianly not for the near-future. As many of you have said, the greatest obstacle here are the pieces of inline asm. I must admit that I'm not very versed in x86 asm, so here too I will be counting on your help.


1. What is the reason to keep compatibility with 2005 runtime when you can target XP with 2008/10/11 just fine (11 might be having some problems in wine though)?
It is more like, there is no reason to drop its support yet. Right now the most I could use from later VS versions is support for the C++11 STL (shared/unique_ptr, atomic, threading), but these are only available in VS2012, so that would be too limiting. This is also the reason I'm considering adding Boost as a dependency, to not have to rely on VS2012 yet still be able to make use of these libraries.


2. What exactly breaks AvsPmod compatibility? This problem is really important since it renders the project useless for a lot of people.
To be able to specify autoload directories in scripts, I had to delay autoloading plugins to the latest point possible. (As a comparison, upstream Avisynth autoloads plugins as soon as the ScriptingEnvironment is initialized, way before any script is loaded.) The result is that, in Avisynth+, the list of plugin functions is not available until a script has been loaded, so for now I have removed exporting special variables like $PluginFunctions$, $Plugin!...!Param$ and $InternalFunctions$, because they didn't make too much sense with the modifications in place. I suspect AvsPmod tries to read these, which is where the conflict comes from.

The best way to solve this would be to (both needed)
- Make AvsPmod only read these variables once a script has been loaded (so that autoload directories that were specified in the script can be taken into account)
- and add the ability to Avisynth+ to force autoloading from the standard directories (to faciliate writing new scripts). This is easy to accomplish without extending the binary interface.

The above two points require modifications to both Avisynth+ and AvsPmod.

[ultim]

Just fixed in current git.

The $InternalFunctions$ and $PluginFunctions$ global vars are gone since the plugin management rewrite. Was that by mistake?

Ah cool, I just saw your post after posting my own. Would you be okay with modifications to AvsPmod? That means you might need a small path to recognize you are using Avisynth+, but I think it might even be posisble to find a unified way that works with both Avisynth and Avisynth+. I'll give some thougth to it if you are interested. Ofc I'll make the necessary modifications too.

[ultim]

Regarding porting to x64:

Taking code from Avxsynth and Avisynth64 could indeed be very helpful, but one must be careful becuase these have been forked long ago and are not up to date with Avisynth 2.6. Meaning the funcationality and semnatics of the filters might have diverged.

x64 VC++ does not support inline asm at all, which means handoptimized pieces of code will either need to be compiled separately using an external assembler, or they need to be converted to intrinsics. IMHO, intrinsics are way better. First of all many/most intel-style intrinsics are portable between compilers (even between GCC, and MSVC), and second, having to compile external .asm files using a dedicated assembler would complicate the build process a lot. Not to mention CMake ATM doesn't even have support for that when generating VS project files. So, TurboPascal7, if you con really help porting to intrinsics, that would be more than great.

Porting to x64 could even be made step-by-step. Most ASM resides in builtin filters, so as long as the core without filters can be compiled in x64, we can be adding more and more builtin filters back to an already working x64 version as the porting continues.

As a separate note, I see a potential problem of maintaining a unified public avisynth.h header without breaking 32-bit plugins. We could ofc always maintain two totally separate versions of that header, but if possible we should try to avoid that.

[TurboPascal7]

There are some problems with intrinsics though:
1) I'm not sure how well they are supported in vs2005 and 2008. While 2008 is probably okay, 2005 might have some serious issues. I haven't worked with it for years and didn't do any C++ back then so I'm not sure.
2) VC++ doesn't support MMX intrinsics in x64 mode. So one either #ifdefs every MMX routine or drops MMX optimization altogether (I do approve dropping it but I'm not sure how many avisynth users don't even have a P4 these days).

[jpsdr]

If ASM reside most in built-in filter and not in the core, maybe what's allready done with avisynth64 could be used, assuming that differences with actual version are more in the core than filters...

[ultim]

1) I'm not sure how well they are supported in vs2005 and 2008. While 2008 is probably okay, 2005 might have some serious issues. I haven't worked with it for years and didn't do any C++ back then so I'm not sure.
Well, I was supporting VS2005 because there was no need to drop it. If it does not have proper support for the intrinsics we need (thereby stepping in the way of 64bit support), that'd be enough reason for me to drop it. But by looking at this (http://msdn.microsoft.com/en-us/library/kcwz153a%28v=vs.80%29.aspx) page, VC++ 2005 does seem to have intrinsics support for SSE2. And if not, I'm okay to drop it.


2) VC++ doesn't support MMX intrinsics in x64 mode. So one either #ifdefs every MMX routine or drops MMX optimization altogether (I do approve dropping it but I'm not sure how many avisynth users don't even have a P4 these days).
We could #ifdef around those parts for 32-bit users on old machines. In the x64-builds, that wouldn't matter anyway coz the presence of SSE2 is guaranteed.

I think I need to add a dynamic dispatch ability to Avisynth+ that filters and plugins can make use of ;)

[TurboPascal7]

One more important question you are aware of is alignment. Did you decide to keep it the way avisynth works now or provide any reasonable guarantee?

I'm not sure that you need to add anything on the core level to help dispatching. Providing a method to get CPU flags (the way it is now) is enough to make filters implementation as simple as possible. I'm not sure how you could simplify it even more.

[aegisofrime]

I'm not a developer so I can't offer any help, just a normal user, but I just want to say thank you to ultim for initiating this project, as well as other contributors. This is probably the most exciting thing on the Avisynth in my few years of video processing. :thanks:

[qyot27]

We could #ifdef around those parts for 32-bit users on old machines. In the x64-builds, that wouldn't matter anyway coz the presence of SSE2 is guaranteed.
I'm definitely in favor of ifdeffing around it. My main setup is a Pentium III-era Celeron, and this was a point of contention that arose in VapourSynth (http://forum.doom9.org/showthread.php?p=1601345#post1601345). The stated solution: use AviSynth if you've got an ancient CPU (http://www.vapoursynth.com/2012/11/minimum-requirements/).

The alternative would be to just disable asm completely on non-SSE2 CPUs, but if there's benefits to be had from MMX and SSE I wouldn't throw it out completely.

[ultim]

I'm definitely in favor of ifdeffing around it. My main setup is a Pentium III-era Celeron, and this was a point of contention that arose in VapourSynth (http://forum.doom9.org/showthread.php?p=1601345#post1601345). The stated solution: use AviSynth if you've got an ancient CPU (http://www.vapoursynth.com/2012/11/minimum-requirements/).

The alternative would be to just disable asm completely on non-SSE2 CPUs, but if there's benefits to be had from MMX and SSE I wouldn't throw it out completely.

I do agree with Myrsloik that programming for Pentium3 is not worth it nowadays, but I see no harm in keeping existing code there for it. All we need to do is put a define around it so that the compiler ignores it in 64-bit builds, while it can still build in 32-bit versions. This works at least for "stable" filters. Others could have a problem, that when they are updated, the filter maintainer would have to update the MMX routines too, and there probably won't be too much motivation for that.

[ultim]

One more important question you are aware of is alignment. Did you decide to keep it the way avisynth works now or provide any reasonable guarantee?
Some changes that I proposed for inclusion in my very first changeset include a refactor of alignment-related functionality. Let me try to make a very-very cautious statement, that I think a minimum alignment guarantee of 32 bytes should be there, unless a script uses unaligned crop.

Getting rid of unaligned crops would undoubtably strengthen this semi-guarantee, but I also must add that there might be other places I have overlooked. Should you find any violation of this alignment, I'll get rid of it for sure as soon as I get to know of it.

[ARDA]

x64 support is also a goal, but certianly not for the near-future. As many of you have said,
the greatest obstacle here are the pieces of inline asm. I must admit that I'm not very versed
in x86 asm, so here too I will be counting on your help.


x64 VC++ does not support inline asm at all, which means handoptimized pieces of code will either
need to be compiled separately using an external assembler, or they need to be converted to intrinsics.
IMHO, intrinsics are way better. First of all many/most intel-style intrinsics are portable between compilers
(even between GCC, and MSVC), and second, having to compile external .asm files using a dedicated assembler
would complicate the build process a lot. Not to mention CMake ATM doesn't even have support for that when
generating VS project files. So, TurboPascal7, if you con really help porting to intrinsics,
that would be more than great.


Porting to x64 could even be made step-by-step. Most ASM resides in builtin filters, so as long as the core
without filters can be compiled in x64, we can be adding more and more builtin filters back to an already
working x64 version as the porting continues.


VC++ doesn't support MMX intrinsics in x64 mode

Some fast thoughts not ordered by priorities;

To compile assembly code with an external assembler is not so hard, at least according to my experience
Nasm/Yasm syntax is very near inline asm, except some pointers syntax, not too much.
They are also portable between enviroments and platforms if I am not wrong. Well, I work almost always
with Microsoft Visual Studio C++ 2008 and Yasm or Nasm.

The idea of separate many filters out of core to port them to x64 is a good one, and for 32
bit version as well. I always dreamt with a core very light, the lightest possible, and different
projects of filters grouped by their functions, tweak colors, editing, resizing, etc.
It could help to bring more developers with some specific knowledge that don't dare to manage the whole
project, besides that it could make easier to track bugs in the core or in a specifig group of filters.
Many years ago I posted something similar, but it seems there is a limit number of dll that can be loaded.
http://forum.doom9.org/showthread.php?p=698052#post698052

In short if possible I wouldn't include them back to core.

I also think there is a lot work previous port to x64, and I agree is not for near future.
And in that future probably mmx or Isse will not be needed anymore, while 32 bits it doesn't hurt
to mantain mmx and mainly Isse cause the profit of non temporal store instructions.

[qyot27]

I don't think it's worth focusing on computers as old as mine either. It's more that I really couldn't see why a simple --disable-asm option wouldn't have sufficed if running the asm on a non-SSE2 CPU would cause problems. I just couldn't grasp how whether the core could use X certain SIMD set would matter to a plugin author, because how I understand the way this all works, the plugin would simply use the desired core function at whatever level of optimization was there. If it was boosted by SSE2 or whatever, great, if not, then it would fall back to the plain C or C++ version and if the user's on a computer old enough for that to be true, then they already know how slow their computer is, so I still don't see the problem in merely letting them run it in an unoptimized state.

[Groucho2004]

I always dreamt with a core very light, the lightest possible, and different projects of filters grouped by their functions, tweak colors, editing, resizing, etc.
This would be the ideal scenario in my book as well but I don't know how much work would be involved to dig out the core and then add these - let's call them native - filters.

[TheFluff]

Well, I was supporting VS2005 because there was no need to drop it. If it does not have proper support for the intrinsics we need (thereby stepping in the way of 64bit support), that'd be enough reason for me to drop it. But by looking at this (http://msdn.microsoft.com/en-us/library/kcwz153a%28v=vs.80%29.aspx) page, VC++ 2005 does seem to have intrinsics support for SSE2. And if not, I'm okay to drop it.

VC++2005 supports it but the code generation apparently has some minor issues, and MMX is worse off than SSE. Avery Lee has written about intrinsics in VC++ at some length, see for example:
http://virtualdub.org/blog/pivot/entry.php?id=46
http://virtualdub.org/blog/pivot/entry.php?id=301
http://virtualdub.org/blog/pivot/entry.php?id=340

[TurboPascal7]

If it was boosted by SSE2 or whatever, great, if not, then it would fall back to the plain C or C++ version and if the user's on a computer old enough for that to be true, then they already know how slow their computer is, so I still don't see the problem in merely letting them run it in an unoptimized state.
But this requires filters/plugins to have this unoptimized plain C version, which might not always be the case (take a look a sangnom2 for example, which doesn't have a plain C version. Original sangnom didn't either). I dunno about you, but I don't really see any point in adding that only to support some 15-years old CPUs (readability problem aside). This is probably not the problem for internal filters though, since most (all?) of them do have the C path.

Guarantees is a good thing and IMHO requiring a newer-than-10-years-old CPU if this makes plugin authors' lifes easier is reasonable.

This would be the ideal scenario in my book as well but I don't know how much work would be involved to dig out the core and then add these - let's call them native - filters.
The core filters are separated quite well and moving them to an external dll wouldn't be hard.

That said, I can't say I fully support the idea. I believe the core itself must provide basic video operations the way it does now and I don't see how separating them would help anything. Especially when compilation of the core is dead simple and fast now.

Some changes that I proposed for inclusion in my very first changeset include a refactor of alignment-related functionality. Let me try to make a very-very cautious statement, that I think a minimum alignment guarantee of 32 bytes should be there, unless a script uses unaligned crop.

Getting rid of unaligned crops would undoubtably strengthen this semi-guarantee, but I also must add that there might be other places I have overlooked. Should you find any violation of this alignment, I'll get rid of it for sure as soon as I get to know of it.
Here we come to the problem I originally tried to explain to IanB (and failed) - only strong guarantees are good for developers.
Take the core filters for example. Right now, most of them are optimized for MMX only, which does not require any type of alignment the way SSE2 do. While porting to x64 and SSE2 (which is the only option if we use intrinsics), one will need to take alignment into account. And without any strong guarantee, he would essentially need to either
1) Template almost every SSE2-optimized routine ever to use unaligned or aligned loads based on provided data. This becomes even worse if we use external asm because simple templates are much easier than any kind of asm macros (https://github.com/tp7/masktools/blob/6723f85c777e12a0877c5ee5fb0b513320f9c13b/masktools/filters/logic/functions.asm).
2) Always use unaligned loads which makes every single SSE2-optimized filter slower only to provide faster crop and possibly faster source filters which I'm not aware of.

That said, I agree we shouldn't make rush decisions. This is indeed a hard question that might (and probably will) break some older filters. I guess I'll do a build with alignment parameter to NewVideoFrame ignored and see how much breaks later.

----
Also, setting up an IRC channel on freenode/rizon or something to speed up the communication process would be nice, I guess (there's already #avisynth on freenode btw).

[ultim]

Separating *all* the filters out of core is not my plan either, actually. I already made external plugins out of many functions that IMHO do not belong into the core, like DirectShowSource, audio timestrething, support for vdub filters etc, and maybe I'll do the same with a few more, but I think too that the core library should be usable on its own too. When I spoke about a step-by-step porting process with a growing set of x64 filters, I was thinking of #ifdef-ing around filter code and gradually enabling them in builds as porting advances.

I *do* support moving seldomly used filters out of the core, so reviewing what belongs there and what not could go onto the TODO list, but I too don't think that the core sould be totally bare and empty of filters. Moving them out, as TurboPascal7 mentioned, is not hard. It is actually so easy that it could be an introductory exercise for anybody wanting to contribute.

ARDA mentioned a DLL loading limit in Avisynth: in the most recent version of the non-plus Avisynth, it is 50. This limit has already been removed from Avisynth+. It was artificial and not-needed in the first place, and it probably got introduced because people were compiling plugins with a static CRT, in which case Windows imposes a limit of 128 loaded DLLs (at least that's the observable effect, the true reason lies elsewhere, but that's a different story). Anyway, my point is, as long as our plugins are linked against a dynamic CRT, having many DLLs should not pose a problem.

[ultim]

Here we come to the problem I originally tried to explain to IanB (and failed) - only strong guarantees are good for developers
I understand and agree with you fully. The reason I didn't give a strong guarantee is not because I don't want to introduce one, it is because I'm not sure that it is currently fully fulfilled. I also want a proper alignment guarantee to be there, which is why I'm going to correct any left-over pieces of code that violate this, if any is still found. I think avoiding unaligned crops brings pretty far in Avisynth+, and definetely further than plain Avisynth.


Also, setting up an IRC channel on freenode/rizon or something to speed up the communication process would be nice, I guess (there's already #avisynth on freenode btw).
I gues we could. Actually I am regularly on rizon anyway (and when not, you'll find my bouncer), but because there are more people accustomed to the forums than to IRC, I find a forum a better place for discussions. If we used IRC, many wouldn't be able to follow.

[ultim]

On the intrinsics vs. asm topic, I will only say that intrinsics are more favorable to me personally, but I'll certainly accept contributed code written in asm too. I won't reject the integration of asm files just because they are not using intrinsics, as long as the contributor also provides a working integration into the current build system. So I say if the contributor is indifferent about using intrinsics or asm, please use intrinsics, but your contribution will be greatly appreciated otherwise too.

Intrinsics integrate more naturally into C compilers and into the existing build system. But maybe most importantly, pure intrinsics can be uniformly compiled into both 32bit and 64bit code, so they win by large in maintainability too. Many prefer asm because they find it easier to read, that is also important, but it is subjective, so I'll leave the choice up to you.

True, intrinsics do not always generate optimal code, but it will be close, and it will still be much better than traditional C code. And let's be honest, hand-written ASM is not guaranteed to be optimal either. Not to mention, code generation from intrinsics improves with better compilers.

But anyway, if you don't want to use intrinsics, that's fine by me too.

[ultim]

What kind of for-construct would you like to see?
Gavino's "for" looks like
for ( variable = init , limit , step ) {
statements
}
where limit and step must evaluate to integers and are only evaluated once. There is also some built-in trickery whether the loop is up- or down-counting (see original thread (http://forum.doom9.org/showthread.php?t=147846) for reference).

The alternative would be a C-style loop, common not only to C but most other computer languages.
for ( statement1 , condition , statement2 ) {
statements
}
which strictly speaking is just a short form for

statement1
while ( condition ) {
statements
statement2
}


I can see upsides and downsides too to Gavino's version. It might be easier to grasp for non-programmers, and also it gives Avisynth some possibilities to recognize infinite-loops, and helps prevent the user from shooting himself in the foot. C-style for-loops could then be emulated using manually written while-loops. On the other hand, it is less generic, non-standard, and despite all the simplifications and built-in automatic, it still cannot prevent all user errors, like when the loop variable is modified inside the body. Programmers could also argue that it is counter-intuitive, because 'limit' and 'step' are only evaluated once even if their value is dynamic.

So, my question is simple: Do you want Gavino-style for-loops or C-style for-loops in Avisynth+?

[mirkosp]

It's not like they are needed, since they can already be done through recursion if one needs to.
Let's not forget that AviSynth does NOT exist as a normal programming language, but more as a NLE tool; I doubt this nature should change with a fork. As such, common programming structures don't really need to be defined.
If one really needs to employ many loops for a script, it's likely because they need to write something complex, and it would be better to straight out write that in a proper language and build a .dll for use.

[wOxxOm]

What kind of for-construct would you like to see?The simpler the better. I like Gavino's with even less 'noise': for var = init, limit, step { .... } #step could be negative.

It's not like they are needed, since they can already be done through recursionRecursion is not human readable. Making dlls isn't worth the effort in simple cases. Using vapoursynth isn't always an option. And there's nothing wrong to have a small set of simple human readable block statements in avisynth-plus. Also, this topic was discussed like a thousand times.

[Sapo84]

It's not like they are needed, since they can already be done through recursion if one needs to.
Which is harder to write, read and probably less efficient (not that it matters).

Anyway, I'd go with the C style for (and no while, for already covers it).
I find the readability of Gavino's one a bit of a problem (if you don't use named parameters), and while his solution is a lot more in line with Avisynth synthax I still prefer having full statements.

[jmartinr]

+1 for C-style

[Gavino]

I can see upsides and downsides too to Gavino's version. It might be easier to grasp for non-programmers, and also it gives Avisynth some possibilities to recognize infinite-loops, and helps prevent the user from shooting himself in the foot. C-style for-loops could then be emulated using manually written while-loops. On the other hand, it is less generic, non-standard, and despite all the simplifications and built-in automatic, it still cannot prevent all user errors, like when the loop variable is modified inside the body. Programmers could also argue that it is counter-intuitive, because 'limit' and 'step' are only evaluated once even if their value is dynamic.
My goal with for was to provide a simple looping construct without the need to use recursion, making life easier for script writers, especially for non-programmers. The syntax is simple so that errors can be quickly recognised and reported clearly.

I made limit and step fixed as I felt they are the parameters that define the loop.
Anything more complex can be programmed as a while loop.

[TurboPascal7]

I'm not sure for loop is required either, maybe because I haven't seen a single useful example of it used in gscript, that couldn't be rewritten with recursion with maybe a bit more effort. If anyone could provide such an example (preferably used in a real working commonly-used script), I would be grateful.

That said, if you feel like adding features to the language just to simplify a little more than zero scripts - sure, go ahead. It's not like it will make the language worse. I don't have any particular opinion on which version is better, they are both fine.

and probably less efficient (not that it matters).
Efficiency should be pretty much identical because after building the filter graph it's all the same.

[Gavino]

I'm not sure for loop is required either, maybe because I haven't seen a single useful example of it used in gscript, that couldn't be rewritten with recursion with maybe a bit more effort.
Clearly it's not required, as any for loop can be rewritten using recursion.
Its purpose is to avoid that 'bit more effort' for the majority of script writers.

[TurboPascal7]

Gavino, I understand and I would fully agree with you in an ideal situation. I just think it would be better to check if the actual demand is high enough (with examples) rather than simply adding features that "would be cool to have". It's never late to add the feature but it will be really hard to remove it. Just look at C++ right now.

[ultim]

I myself find the new block-if the most useful, but the looping constructs are useful too. Please don't turn this discussion into whether to have looping constructs at all or not. The fact that previously loops have been emulated using recursion proves that loops are needed and sought after in Avisynth. Especially in a language that is designed to be simple to write and simple to interpret by humans, "for" or "while" can mean a lot, because they are way easier to program and read than even the simplest form of recursion.

So there will be ifs, fors and whiles (and admittedly I'll also see if I can hack a "break" statement into the language). If you are more comfortable with recursion, you can still keep using it. Now please, let's get back to the question at hand and let's vote which "for" variant you'd prefer.

Edit: I understand that none of these new constructs are technically needed, as the same effect can be achieved using other methods. However, those methods make scripts harder to write and to read. These new constructs are improvements, and improving the existing functionality of Avisynth is the goal of Avisynth+ after all.

[qyot27]

But this requires filters/plugins to have this unoptimized plain C version, which might not always be the case (take a look a sangnom2 for example, which doesn't have a plain C version. Original sangnom didn't either). I dunno about you, but I don't really see any point in adding that only to support some 15-years old CPUs (readability problem aside). This is probably not the problem for internal filters though, since most (all?) of them do have the C path.
I don't see how it requires plugins to have it. It just means the user is SOL if a plugin doesn't have a fallback, same as they are now if they try to run a plugin with incompatible SIMD on their machine. But I don't see how the core running unoptimized has an effect on what the plugins support.

I'm looking at it more from the perspective of growth toward other machine architectures - having an unoptimized version maximizes portability. Sure, there could be wholesale additions for NEON or AltiVec or SHmedia or whatever else, but that'd still have to be a case-by-case basis.

[kypec]

ultim, you should probably create separate thread with poll about "preferred loop constructs" then as to move it outside this topic. BTW, I'd prefer C-style syntax myself...

[ARDA]

What kind of for-construct would you like to see?
Gavino's "for" looks like


So there will be ifs, fors and whiles (and admittedly I'll also see if I can hack a "break" statement into
the language). If you are more comfortable with recursion, you can still keep using it.

Related with Gavino-style for-loops or C-style for-loops I have no preferences, but obviously the
easier for script writers the better, to make more human readable avisynth language is something to add
to a TODO list, avisynth is far to be intuitive or friendly for normal user. Gavino's work is in that
direction, nor more nor less.(My opinion, never interchanged views with him)
I see you have already decided to conform everybody including both and something else, luckly you
have the energy and skills for everything. Thanks again.



Porting to x64 could even be made step-by-step. Most ASM resides in builtin filters, so as long as the core
without filters can be compiled in x64, we can be adding more and more builtin filters back to an already
working x64 version as the porting continues.


Separating *all* the filters out of core is not my plan either, actually.
but I too don't think that the core sould be totally bare and empty of filters.


I always dreamt with a core very light, the lightest possible, and different
projects of filters grouped by their functions.......

I do not see the word empty or *all*, maybe is my poor english, but I remarked the lightest possible,
that I don't define with precision, so I left opened to a research, study or discussion.


To compile assembly code with an external assembler is not so hard, at least according to my experience
Nasm/Yasm syntax is very near inline asm, except some pointers syntax, not too much.
They are also portable between enviroments and platforms if I am not wrong. Well, I work almost always
with Microsoft Visual Studio C++ 2008 and Yasm or Nasm.


True, intrinsics do not always generate optimal code, but it will be close, and it will still be much better than
traditional C code. And let's be honest, hand-written ASM is not guaranteed to be optimal either. Not to mention,
code generation from intrinsics improves with better compilers.

I do not say asm is better than intrinsics, just remark it is not so hard to move inline asm to Nasm/Yasm.
And is also portable, probably to more platforms.
Obviously you have to work under and enviroment that can use Nasm.
Asm code can be non optimal of course. I was thinking about the task of moving out inline code you already have
in built in filters nowadays, optimal or not, while preserving many mmx and Isse code.
There are some Isse codes that are well optimized in avisynth and it is very difficult to get a SSE2 or higer version
to work better, mainly when you have to work with misaligned ptrs.


I also think there is a lot work previous port to x64, and I agree is not for near future.
And in that future probably mmx or Isse will not be needed anymore, while 32 bits it doesn't hurt
to mantain mmx and mainly Isse cause the profit of non temporal store instructions.

@ultim
All these are just some quick ideas, I see you are working hard with other issues, more immediate needs
and improvements so we can leave all this discussion for future.


Thanks for all the work you are doing, and keep on.

[ultim]

Hu, I can rename the thread :) Btw, Wilbert, I never thanked you for moving us out of the Avisynth a4 thread. So sorry for the trouble and thank you.

[TheFluff]

Separating *all* the filters out of core is not my plan either, actually. I already made external plugins out of many functions that IMHO do not belong into the core, like DirectShowSource, audio timestrething, support for vdub filters etc, and maybe I'll do the same with a few more, but I think too that the core library should be usable on its own too.

There are a bunch of "core" filters that should be removed and some that should be restructured, because the current selection is a horrible mess of old cruft and arbitrary ad-hoc "design". I'll quote myself:

SelectEven/SelectOdd are special cases of SelectEvery and are not going to be implemented (you can of course "implement" them trivially in Python if you want). Similarly, ApplyRange is a special case of Trim (plus splicing) and can be trivially implemented in Python as well. Vapoursynth will not have an 1:1 copy of all Avisynth functions, especially not since the scripting language is now a real programming language so implementing interesting script functions is mostly trivial.

Avisynth has a ton of old crufty functions that are in the "core" for very unclear reasons, anyway. Why are there five different level manipulation functions (coloryuv, levels, limiter, rgbadjust, tweak) with a ton of overlapping functionality? Why are things like PeculiarBlend, FixBrokenChromaUpsampling and FixLuminance core functions? Why does CropBottom exist? Why does the basic "fade" function (equivalent to dissolve+blankclip) need nine different function names and prototypes? God (or maybe BenRG) alone knows.

I mean, check the documentation for FixLuminance (http://avisynth.nl/index.php/FixLuminance). Why the hell has this been in the core for over a decade?

Other redundant functions that should be removed:
ShowFiveVersions - exactly equivalent to StackHorizontal with five arguments. Why?
MonoToStereo - has been explicitly deprecated since 2.5 and replaced by MergeChannels
Pulldown - it's SelectEvery with the cycle forced to 5, use that instead
SwapFields - the documentation explicitly says it's just an alias for SeparateFields().ComplementParity().Weave()

[Myrsloik]

I have one comment to make about the inline assembler/softwire/mess that is in avisynth. Most of it is of a horribly low quality.

Replacing it with proper C++ code probably wouldn't slow it down that much. You also need a proper C++ code path just to test that the assembler you wrote is correct anyway. But that's something no other posters seem to take into consideration.

Anyway, interesting to see some kind of development going on.

[jpsdr]

Being a big user of pulldown, i personnaly prefer to keep it. Some are like macro in C. When something can be used very often, it's nice to have a more compact syntax. I prefer writing PullDown(0,2) than SelectEvery(5,0,2).AssumeFrameBased.
Even if i don't use it, showfiveversion has the same purpose. They are like macro in C, made to simplify syntax of something which may be used very often.
Seriously, i prefer SwapField, name is explicit, syntax is simple, than SeparateFields().ComplementParity().Weave() !
Think of someone who simply want to swap fields... Good luck guessing SeparateFields().ComplementParity().Weave() when you don't have the "swapfield" function !!
Same for pulldown.
Personnaly, i think these shouldn't be removed.

[TurboPascal7]

jpsdr
All that can be solved by shipping a single avsi script with the core, containing all the removed functions.

[ultim]

All that can be solved by shipping a single avsi script with the core, containing all the removed functions.
I'm not sure putting them into .avsi scripts is how much better then being included in the core native filters. Even if in the core, we can cleanly separate them out into their own filters/shortcuts.cpp file (or whatever the name) to not pollute the main filters. But putting them into a separate script incurs a needless startup cost (the'd be autoloaded anyway, but now they'd need to be separately loaded, parsed and evaluated).

[TurboPascal7]

ultim
I'm not saying they should be definitely separated, just stating that the issue can be solved without any real affect on the user.
I think both solutions are fine: the cost of loading a single avsi is not that high, but having them in a separate cpp file would not be bad either, especially since most of them can be implemented in one-two lines.

[TurboPascal7]

While we're at the question of removing stuff: is there any reason TemporalSoften's mode 1 still exists?
Documentation says mode 2 "has a new and better way of blending frame and provides better quality. It is also much faster." and it looks to me almost everyone is using it anyway, so why keeping it at all?

[ultim]

Avisynth Plugin Writing Tips #1: Exceptions

Exceptions thrown from a module should only be caught in the same module. Otherwise you can experience weird and hard-to-debug errors in the plugin. Not adhering to this advice will result in code that can sporadically fail, or work on your computer consistently but fail on other machines.

Unfortunately, avisynth.h contains the AvisynthError class, giving plugin authors the false impression that it is safe to throw and catch these exception objects. It is not. The problem is not in the definition of this class, but in the implicit encouragement to throw C++ exceptions across DLL boundaries. Here are some tips to avoid getting caught in the deepest pits of hell:
- When throwing exceptions on your own, it is best not to use AvisynthError. Not using it will stop you thinking that AvisynthError has some special meaning, or that it can be used to throw to (or to catch from) avisynth.dll.
- Exceptions thrown by you should always be caught inside your plugin. You should not let exceptions propagate outside of your DLL (unless thrown using ThrowError), to Avisynth.
- Errors thrown by Avisynth should not be caught by you. In specific, don't wrap calls to Avisynth in try-catch blocks, because you cannot rely on it working correctly in every situation. If you need to detect errors, validate user parameters in your plugin, or use other API facilities provided by Avisynth, like IScriptEnvironment->FunctionExists().
- If you want to throw an exception to the user and/or to Avisynth, then only use IScriptEnvironment->ThrowError(). You should not call C++'s "throw" yourself for this purpose (see 2. point), and you should not catch the error thrown by ThrowError() yourself (see 3. point).
- If you want to catch an exception, want to do something based on that and finally raise an exception to Avisynth, don't rethrow. Catch your own exception (unless thrown by ThrowError), then call ThrowError separately.

Ignoring the above tips can still result in a fully working binary, but that is only guaranteed under very specific circumstances, more specifically when you've compiled your plugin with the *exact* same compiler version as the avisynth.dll was compiled with, AND when linking to the CRT runtime dynamically. Given that plugin authors can use whatever compiler they want, and that an avisynth binary can be supplied by any community member, it is unwise to rely on such detail.

These tips apply to all Avisynth versions (e.g. to 2.5 and to 2.6, to "classic" Avisynth and to Avisynth+, etc).

[ultim]

qyot27: I looked at the avisynth_c.h you mentioned that has support for the new colorspaces in 2.6. What sucks is that it is neither source- nor binary-compatible with the avisynth_c.h header from upstream avisynth. So I have to the choice of 1) either keeping the original, or 2) merging features from ffms2 but breaking C-compatibility to upstream.

As a sad note, the C-interface has a bunch of baked code. What surprises me is that Avisynth now has three different APIs (2.5, 2.6, and C), and all of them suck. I honestly feel tempted to finally create an API for Avisynth that 1) doesn't rely on the representation of internal data structures, 2) doesn't rely on baked code, and 3) can be used universally from many different compilers and computer languages (right now whichever existing interface you choose, you can pick at most only one of these properties). I'm just afraid that XKCD #927 (http://xkcd.com/927/) will set in.

[Groucho2004]

I'm just afraid that XKCD #927 (http://xkcd.com/927/) will set in.
Or this (http://en.wikipedia.org/wiki/Survival_of_the_fittest). :D:D:D:D

[qyot27]

qyot27: I looked at the avisynth_c.h you mentioned that has support for the new colorspaces in 2.6. What sucks is that it is neither source- nor binary-compatible with the avisynth_c.h header from upstream avisynth. So I have to the choice of 1) either keeping the original, or 2) merging features from ffms2 but breaking C-compatibility to upstream.
As it was a partial merge of 2.6 features into it, I'm not surprised (compatibility with the upstream's 2.5-centric C interface was relegated to a secondary header that defined old versions of avs_get_row_size_p and avs_get_height_p). The intent was probably for the upstream to catch up to these features whenever 2.6 finalized, or something.

Having a new API is likely going to be needed regardless, since the introduction of really-new or novel features would probably break the existing one anyway. Might as well develop the new features against an API that's cleaner and easier to maintain, IMO. It'd also be an opportunity to document it better, if all the complaints I've heard over the years about how poorly documented the existing one is is any indication (noble efforts to rectify this notwithstanding (http://forum.doom9.org/showthread.php?t=162565)).

[Wilbert]

qyot27: I looked at the avisynth_c.h you mentioned that has support for the new colorspaces in 2.6. What sucks is that it is neither source- nor binary-compatible with the avisynth_c.h header from upstream avisynth.
Why don't you create one for upstream avisynth with the baked code removed? I'm sure IanB is interested in it.

[ultim]

Why don't you create one for upstream avisynth with the baked code removed? I'm sure IanB is interested in it.

The largest benefit of that would be so that compatibility to existing C plugins can be kept, like it was done with the 2.6 interface. But that wouldn't solve all the problems. The C interface has multiple structures which were layed out exactly as their C++ counterparts, and they in the end represent core structures, and both Avisynth as well as plugins can depend on their layout. This means it is impossible to modify these types without the possibility of breaking past and future plugins. So even if I removed the baked code, all we'd gain is the ability to modify method definitions. Which is better than nothing I suppose, but I refuse to build a 4th! interface that still has some of the same fundamental problems as the previous three.

That being said, I'm not working on any completely new interface right now. If at all, building a new interface will come later when there is a definite need for it for a feature that is being worked on. But that doesn't mean we cannot think about it until then.

For now, all the new methods that I'm adding to the ScriptEnvironment class are ending up in an "IScriptEnvironment2" abstract class. This has the benefit of never conflicting (layout-wise) with methods that IanB might be inserting into IScriptEnvironment, and it also keeps full compatibilty with existing plugins. Basically, it gives me a nice playground without any risk of breaking plugins until it stabilizes. However, because it is still continuously changing, it is only safe to use IScriptEnvironment2 from inside Avisynth+, or from plugins built and distributed with Avisynth+.

I'll have a real problem though if I ever see the need to modify VideoFrame(Buffer), AVSValue, IClip and so on, thanks to the limitations of the current interfaces. Unlike ScriptEnvironment, these aren't centered around the concept of abstract interfaces like COM-techniques would require.

[ultim]

I have just updated the online repository and the binaries in the first post with a new version, compiled fresh today. There are some goodies here, so let's see:

- First, Gavino's scripting extensions have been integrated. Be sure to say thanks to Gavino for his work again. I've only made minor modifications to his patches, like properly handling empty "if" blocks or missing optional "else" parts, and taking the "last" variable from before the new "if", "while", and "for" statements better into account. I've also added a "break" statement that will allow you to jump out of any loop without reaching the terminating condition. I've counted the votes for the "for"-style carefully, and your votes turned out equally even, so in the end I picked Gavino's original simplified style. You can still write any kind of sophisticated loop using "while".
- I added back proper AvsPmod support, that got temporarily removed earlier 'coz of the plugin system's rewrite. Built-in functions will now work as before, but to get autoloaded functions to show up in AvsPmod will need a slight modification to AvsPmod. This is unfortunately necessary, because autoloading has to be delayed to support adding search folders in scripts. As soon as AvsPmod gets modified you'll have full functionality back again. The needed modifications won't interfere with traditional Avisynth.
- The C interface is now probed before the 2.5 plugin interface, making ffms2 work again even if you're not using qyot27's latest build.
- The "crop" function now defaults to aligned crop. You can still controll alignment using its second parameter, but if you omit it the default is now for the new frame to be aligned. This is important for plugin authors so that they can have a stronger alignment guarantee, in the end leading to faster processing in multiple plugins.
- And as always, there are cleanups and refactors, in an ongoing effort to make the sources higher quality. Not as many as in previous releases, but still. Of course more to come in the future.

64-bit support
There's actually one more feature item missing from the above list. The archive in the first post includes both 32-bit and 64-bit builds. All ASM from the core got replaced with C-code or intrinsics, the inline ASM in filters got sandwiched in ifdefs, and I fixed up any remaining issues that prevented the core from working correctly. But before rushing to download to run your uberscript in 64-bits, please note that no porting of the builtin filters has been done yet, making that build hardly usable. You'll find that many essential filters are missing, like resizers and color space conversions, just to name a few. So trust me, as a user you are most likely better off using the 32-bit build for now. But the 64-bit is there for the adventurous, for motivated testers and developers, and for all those who are wishing to help port the missing functionality to 64-bit. Besides, the new 64-bit build is compatible with existing 64-bit plugins found here (http://code.google.com/p/avisynth64/wiki/PluginLinks), here (http://forum.doom9.org/showthread.php?t=152800) and here (http://members.optusnet.com.au/squid_80/), so you might actually be able to use the 64-bit version for something if you don't rely much on internal filters.

So, there is a working 64-bit build, though farly gutted out. If you know some intrinsics, please consider helping out with the port, even if with only one or two routines. Even without knowledge of intrinsics or ASM, if you can rewrite some algorithms that were only available in ASM before into plain C, that would already be a lot of help. Feel free to start anywhere you like, and rest assured, I'm also continuing my work on Avisynth+.

Edit: Oh, I almost forgot. I added a small number of issues to the GitHub page. They are just ideas and enhancement requests. If you feel like it, you can also pick up something from there if you prefer working on something new, or you can also add your own ideas.

[StainlessS]

I've also added a "break" statement that will allow ...

thumbs up on that, thanx. EDIT: Indeed, lots of good ideas, GScript & Grunt have been ripe for inclusion for eons. :)

[kypec]

Hi ultim,

today I've pulled the trigger and wanted to give your build a try but there seems to be something missing or broken in Windows build avsplus_2013_10_15.zip.
I'm using Windows 7 64-bit version and all my scripts were working correctly with AviSynth 2.60 build May 16, 2012.
As soon as I replaced dynamic libraries in my SysWOW64 directory with files from your package (placed under x86 folder) AviSynth.dll and DevIL.dll my script that uses MCTD plugins stopped working, giving this error message:
AVSMeter 1.7.0 [AVS2.6] by Groucho2004

TemporalSoften::GetFrame is not yet ported to 64-bit.
Here's my script that works fine if MCTD processing is commented out:# Initialize variable with path to my A/V tools
plugindir="C:\PROGRAM FILES (x86)\AVTools\plugins\"
# Load all the plugins that will be used in the script
LoadPlugin(plugindir + "DGDecode.dll")
Import(plugindir + "MCTD_Loader.avsi")
# Open source video clip
MPEG2Source("D:\Source\That70sShow\S1E01.d2v", info=3)
# Crop to get rid of tiny overscan and ugly artifacts at the edges
Crop(2, 2, -2, -2)
# Denoise - must be left out to make it work with AviSynth+
MCTemporalDenoise(settings="low", stabilize=true, enhance=true, edgeclean=true, sharp=false)
# Add black borders to get standard mod16 resolution 720x576
AddBorders(2, 2, 2, 2)


Could you please investigate why internal filter TemporalSoften is throwing this misleading error? AFAIK I'm not using 64-bit of AviSynth, I've never had it installed on my system and unless you mixed DLLs in your package I can't see why standard 32-bit should fail on me...

Thanks in advance! ;)

[vdcrim]

TemporalSoften::GetFrame is not yet ported to 64-bit
It's a typo (https://gist.github.com/vdcrim/bf78785e0a59bea6bd2f).

[kypec]

It's a typo (https://gist.github.com/vdcrim/bf78785e0a59bea6bd2f).

Excellent! :thanks:
Hopefully ultim uploads new fixed build before I even try to get familiar with git on Windows and compile patched sources myself. :D

[ultim]

I'm at work, but I'll post a fixed build very soon, I'll be home in a few hours. The bug is only in the most recent build from okt.15. and it mistakenly flags TemporalSoften to be unavailable on 32-bits.

[zero9999]

(removed build)

[ultim]

And we also get an ICC-build :) Thanks Zero9999!

But for the sake of completeness (or more like coz I've promised) I also updated the release in the first post. It also contains the fix for the typo ofc, and a small number of other things only important for developers (details in the GitHub commit log).

TurboPascal7: I have not forgotten about alignment.h, changes will come next time.

[Groucho2004]

x86 ICC14 build
That's slower than the VS8 build from ultim. Did you use PGO? Did you read the Intel optimization guides?
Anyway, my experience with Intel compilers is that they get slower with each new version.

[zero9999]

That's slower than the VS8 build from ultim. Did you use PGO? Did you read the Intel optimization guides?
Anyway, my experience with Intel compilers is that they get slower with each new version.

it was a little faster for the first script i tested with, but using a test script with only the internal functions/filters, it's indeed a little slower. I didn't have much luck with PGO so far, it made the build much slower.

[ARDA]

@ultim;
First of all thanks for the great effort you are doing.


That's slower than the VS8 build from ultim. Did you use PGO? Did you read the Intel optimization guides?
Anyway, my experience with Intel compilers is that they get slower with each new version.


it was a little faster for the first script i tested with, but using a test script with only the
internal functions/filters, it's indeed a little slower. I didn't have much luck with PGO so far,
it made the build much slower.

I don't know if this is the right thread, I have no intention to pollute it.
But as we are going to have several and different compiled versions of avisynth around, I would like to arrive
to a common benchmark tool and conditions, so we can compare.

@Groucho2004 and zero9999;
Will you please post your benchmark methods with most possible details ?

Related to Gpo or cpu dispatch options in compilers, I had no good experience with avisynth
in that matter. I think the question is that in many in built functions in avisynth mmx, isse
sse2, sse3 and ssse3 inline asm code cannot be touched by the compiler, so just c++ will be
optimized or cpu dispatched which obviously will improve almost nothing for that cases.
Some years ago I tried with intel compiler 10 but never could get any good result, but remember
also (don't know about newer versions) that in some cases when those codes run in a AMD
cpu were derivated to a general not optimized library.

Thanks ARDA

[TurboPascal7]

There is not much point in compiling avs+ with ICL right now. However it will be "a must" for optimal performance after some code gets ported to intrinsics as VC tends to do a pretty bad job here. I also can confirm that auto-vectorization does not and probably will never help avs core much.

Next, I do not believe that there should be several differently compiled version of avisynth flying around and I would like to ask other people to refrain from posting binaries at least for now as it might just create additional confusion. In the end I believe the best solution would be to provide one binary, built with ICL, as code it generates for intrinsics seems to be faster than VC even on AMD processors. Getting ICL is not hard even if you are not a student.

As for benchmark methods, I think AvsMeter is pretty much the standard tool right now.

[ARDA]

Next, I do not believe that there should be several differently compiled version of avisynth flying around
I would like to ask other people to refrain from posting binaries at least for now as it might just create
additional confusion. In the end I believe the best solution would be to provide one binary,
built with ICL, as code it generates for intrinsics seems to be faster than VC even on AMD processors.
Getting ICL is not hard even if you are not a student.

I second


...but remember also (don't know about newer versions) that in some cases when those codes run in a AMD
cpu were derivated to a general not optimized library.

I was talking about this http://www.agner.org/optimize/blog/read.php?i=49#49

Thanks

[TurboPascal7]

This is not relevant here because avisynth doesn't use any Intel libraries and will not rely on automatic vectorization in the future. Better register allocation and instruction ordering that ICL offers for intrinsics is still better even on AMD processors (but probably not optimal).

[ARDA]

This is not relevant here because avisynth doesn't use any Intel libraries

I don't understand. When you compile with intel compilers, just to mention a few by heart it has it own
memcpy, memset, memmcmp, all strings libraries, etc, etc, etc.
I don't know if that legal question was solved and it is not present in nowadays versions.
I don't want to pollute this thread, so for me this subject is enough discussed.

I'll be waiting Groucho2004 answer about benchmark methods,

Thanks

[Groucho2004]

in some cases when those codes run in a AMD cpu were derivated to a general not optimized library.
This only applies if you use automatic CPU dispatch optimizations (Qax..). I only use Qx... which "hardcodes" the supported instruction set into the binary. For example, if you use "QxSSE2", you need at least a P4 or a Athlon64.
Also, the maximum optimizations don't always ensure maximum performance. It's a lot of work to optimize a binary (especially with the Intel compiler) but sometimes you get pretty decent gains.

[Groucho2004]

I'll be waiting Groucho2004 answer about benchmark methods
Here goes:
I used this (http://www.mediafire.com/?8g500sjm9tr575a) script (which I also use for PGO) and AVSMeter.
The results are (i5 2500K @ 4GHz):
Ultim's latest DLL: 29.73 fps
Official Alpha5 DLL: 28.95 fps
My Alpha5 ICL10.1 DLL: 30.02 fps
zero9999's ICC14 DLL: 24.56 fps

I'm aware that the script does not cover all internal functions and I'd be happy about suggestions to improve it.

[ARDA]

Thanks Goucho2004, I'll work with it and if I have some suggestion I will post here

ARDA

[ultim]

Theoretically there shouldn't be any difference between the runtime performance of Avs+ and Avs yet. Any current differences are probably due to different compilers. While some of my current changes might improve running time marginally (removed allocations, less exceptions), I have not yet done anything that was specifically aimed at increasing performance.

Edit: The only difference in speed should be initialization/loading time, where I did make conscious improvements.

[ARDA]

- The "crop" function now defaults to aligned crop. You can still controll alignment using its second
parameter, but if you omit it the default is now for the new frame to be aligned.
This is important for plugin authors so that they can have a stronger alignment guarantee, in the end leading
to faster processing in multiple plugins.

Allow me please to point some issues.
I know it is not an important subject, but just a warning for plugin developers but mainly for users.
Take following code just as an example.


MPEG2Source("D:\TRABAJO32BITS\MALENA\MALENA.d2v")# 720 x 576, use your source

AvsTimer(frames=1000, name="ANYONE",type=3, frequency=2000, total=false, quiet=true)#use your cpu frequency

#crop(8,72,700,432,align=true)
#or
crop(8,72,700,432,align=false)

#BicubicResize(640,352) # or anyother sizes and resizers, and many filters as well
#or # just two cases I've tested.
flipvertical() # as flipvertical that in fact is just a bitblt

AvsTimer(frames=1500 ,name="ANYONE",type=3, frequency=2000, difference=1, total=false)


Which one of crops do you think is faster? At first sight we should say with align=true, but
it isn't; the fastest one is with align=false, between 3% till 20% faster with resizers and more than
40% faster with filpvertical(). Why?
Simple with crop + align=true in fact there are two movements of the whole frame, first a bitblt of align=true
and second to the new created frame by resize function, or just another bitblit with flipvertical.
Obviuos this situation only will happen if following filter in the chain inmediately after crop creates
a new frame. And from that frame onwards we shall have again an aligned buffer.

There are a few filters that could work in place under the appropiate conditions and in that case
an align=true maybe could be better. I've still some doubts, I didn't test.
(one by heart, Isse code of Tweak by dividee, but probably many others)

So I don't say align=false is better, just that final user should be aware how next filter in the
chain after the crop works. Is that possible? Probably not. At least I should add a WARNING
in documentation for users. IF YOU RESIZE IMMEDIATELY AFTER A CROP, ALIGN=FALSE COULD BE FASTER.

As far as plugin developers, I would never trust that the source (last) you are receiving is always
aligned b16 for sse2 or b32 for new AVX. So I would check always, except that the only posibility
to have misligned data were with crop align=false, I don't remember and didn't check.?

I hope this can be useful. ARDA

[TurboPascal7]

Yes, without a doubt, unaligned crop is faster in some (most) cases. Sometimes significantly so. Should a user know about it? Probably. Does it matter? Not really. You don't use crop often in a single script and even on my Core i7 860 system (stock clocks, 4 years old), aligned crop at 1080p runs over 2000fps. Hardly a noticeable impact for me as my scripts rarely run faster than 1fps in the end.

As for developers - yes, one should always assume that he can get any kind of data. But one also can assume that most of the times data he will receive will be aligned. This means that he can omit unaligned routines completely, dispatching all such cases to plain C implementation.

For example in the core filters recommended dispatch looks like this:
if ((env->GetCPUFlags() & CPUF_SSE2) && IsPtrAligned(srcp, 16)) {
process_sse2(some args);
} else
#ifdef X86_32
if (env->GetCPUFlags() & CPUF_MMX) {
process_mmx(some args);
} else
#endif
process_c(some args);

[Groucho2004]

Hardly a noticeable impact for me as my scripts rarely run faster than 1fps in the end.
Ouch. You must have terrible sources if they require this kind of filtering.

[ultim]

As Turbo pointed it out, what is important is that plugin developers can know that in the majority of the cases, they will get aligned data. Of course they should not rely on this, but they should definitely optimize for it. Crop is a relatively cheap filter, little more than a bitblt, and the performance that is lost by copying a frame one extra time should be lower than what you'd loose if you ran a more complex filter in its unoptimized version.

[ultim]

Here goes:
I used this (http://www.mediafire.com/?8g500sjm9tr575a) script (which I also use for PGO) and AVSMeter.
The results are (i5 2500K @ 4GHz):
Ultim's latest DLL: 29.73 fps
Official Alpha5 DLL: 28.95 fps
My Alpha5 ICL10.1 DLL: 30.02 fps
zero9999's ICC14 DLL: 24.56 fps

I'm aware that the script does not cover all internal functions and I'd be happy about suggestions to improve it.

Out of curiosity, I ran your script through a profiler, and have found that over 40% of time was spent in text antialiasing, more specifically in Antialiaser::GetAlphaRect().

[Groucho2004]

Out of curiosity, I ran your script through a profiler, and have found that over 40% of time was spent in text antialiasing, more specifically in Antialiaser::GetAlphaRect().
I know, it's far from a good source for profiling. It does however have a bunch of functions but apparently their CPU usage pales in comparison to the text rendering. I guess I should just use "blankclip" as a source.

[Mystery Keeper]

Text anti-aliasing as separate procedure? Why don't you use Anti-Grain Geometry library and render text already anti-aliased?

[ultim]

I know, it's far from a good source for profiling. It does however have a bunch of functions but apparently their CPU usage pales in comparison to the text rendering. I guess I should just use "blankclip" as a source.

Basically anything that doesn't dominate over all the other filters this much. Also, maybe text rendering could be kept, only less text/smaller area for text.

Text anti-aliasing as separate procedure? Why don't you use Anti-Grain Geometry library and render text already anti-aliased?
Antialiaser::GetAlphaRect() is just part of the antialiasing. There might be other libraries out there for this purpose (anti-grain, cairo, freetype etc.), but the code for this is not much, it worked just fine, and at least there was no need to link to yet another library. We'll need to link to one when we go cross-platform though, since we're currently relying on GDI for the actual drawing.

[qyot27]

Just because I've been curious about this since the rewritten demuxer was committed this past March, can the 64-bit build of AviSynth+ (or heck, the old builds of AviSynth64) work correctly* with Win64 builds of FFmpeg that were compiled with --enable-avisynth? I have no 64-bit Windows setup to test with, so I've never bothered to actually check.

*obviously defining 'work correctly' in the context of what's actually available in said 64-bit builds of AviSynth(+). Even if it's just Version() or simple file loading, that's enough.

[l33tmeatwad]

Just because I've been curious about this since the rewritten demuxer was committed this past March, can the 64-bit build of AviSynth+ (or heck, the old builds of AviSynth64) work correctly* with Win64 builds of FFmpeg that were compiled with --enable-avisynth? I have no 64-bit Windows setup to test with, so I've never bothered to actually check.

*obviously defining 'work correctly' in the context of what's actually available in said 64-bit builds of AviSynth(+). Even if it's just Version() or simple file loading, that's enough.Yes, it works with both AviSynth64 & AviSynth+ 64-bit.

[ultim]

I'm giving Avs+ a Boost (pun). It has libraries that are really useful for Avs, so much that in fact many have been included in the new C++11 standard from there. I guess you could ask, why not just use the corresponding standard headers then? First, not all are in C++11. But much more importantly, even those that are, are not all supported by Visual Studio, or only supported by Visual Studio 2013. And I don't want to limit developers to use only the newest VS that been released literally a week ago. In the far future, when C++11 support has become widespread enough among people, in another 2-3 iterations of VS maybe, I'll be glad to switch from Boost to the standard headers, but until then, Boost will make sure we get the appropriate support on every compiler. Below is a list of libraries that could prove really usefull. Many of them are non-trivial, but Boost has well-tested, time-proven, maintained, and portable implementations of them.

Boost libs that I've set my eyes on: Atomic, Smart Ptr, Filesystem, Thread, Lockfree.
Other Boost libs that can come in handy: Locale, Unordered, Log, Static Assert, and some others.

[zerowalker]

Just to make sure, it does not speed up scripting itself, meaning for example QTGMC?

[TurboPascal7]

Right now - no. To speed up something like QTGMC one will either need faster plugins or maybe good multithreading. Plus faster plugins.

[zerowalker]

Okay then i got things right, but will this take over the Avisynth development?
Cause seem to be quite many branches going on, as well as Vapoursynth, which however is supposed to superseed Avisynth eventually if i understand things correctly.

[TurboPascal7]

Vapoursynth is an entirely different world that some people are not completely happy about (including the author of this fork (http://forum.doom9.org/showpost.php?p=1644133&postcount=17) and myself for example).

As for other Avisynth branches - Avxsynth and Avs64 are effectively dead. Avisynth MT is alive but it's not a real "branch" of avs and it does not get much development and it won't "superseed" Avisynth.

As for avs+ taking over the avisynth development - this depends on your understanding of "taking over". No one is going to assassinate IanB and I'm pretty sure he will be maintaining the official avisynth branch in the future. Whether the official branch will continue to be the most widespread version is another question. We'll see.

[zerowalker]

I see, had quite some positive thoughts of Vapoursynth as it seems good in itself (Avisynth is old, time to renew so to speak).
But well it´s always more than meets the eye.

Good to know, and as you end up, it´s those 2 i am concerned, IanB and avs+.

Like to have 1 to follow, but guess i will have to wait to see if if one of these will "win" the war, or if it will continue to be 2 independent branches.

Thanks for the info.

[ultim]

Avisynth Plugin Writing Tips #2: Parallel execution

Avisynth-MT and VapourSynth both support multithreading, and it is being implemented in Avisynth+ too. All of them require the same things from your plugin. Here is a list of what you as a plugin author can do to support execution on multiple threads. If you are the author of any Avisynth plugin, please update your filter according to these rules if needed. Doing so will make sure your plugin can execute seamlessly when multithreaded. Furthermore, following these rules will not only guarantee correct execution in multihtreaded environments, it will also provide optimal mulithreaded performance.

Short list for those on the run:
- Unless you have the slowest filter in the world, don't start threads in your plugin.
- Never use global or static variables. In addition, your filter class should only have read-only members which are initialized during construction.
- Don't reuse the IScriptEnvionment pointer between method executions.

And again, the same points with a bit of more explanation:
- In general, do not slice up your frame and start multiple threads on your own. Threading has its own performance overhead, and it is only worth doing it manually if your filter takes a lot of time to execute. And even if your filter is extremely slow (like fft3dfilter), you should try to optimize its single-threaded performance (by using SIMD instructions or choosing a more efficient algorithm) rather then manually threading it. Optimize for single-threaded performance, and as long as you follow the other rules below, you will get automatic and correct multithreading from Avisynth.
- Do not cache frames yourself. You might think it is efficient because you won't have to request/compute them in the next frame, but you are wrong, and there are several reasons why. First, if you write your own cache, you will have to introduce a global state to your filter, which means you will have to take care of synchronization between multiple threads too, which is not easy to do efficiently with caches. Second, keeping copies of past frames also means there will always be multiple references to them, thus Avisynth cannot pass them as write pointers to other filters, and will have to do an extra copy of it more often. And last but not least, Avisynth has a very extensive caching mechanism, and if you request the same frame multiple times (even when you need it for different frame requests), chances are you will get it for free anyway, so your own caching is just pure overhead.
- As a general extension to the previous rule, try not to keep any state between frames. In the optimal case your filter class should only have read-only members which are initialized during construction. Surely this is not always possible with every algorithm, but most times it is, and this is what you should strive for.
- As stated before, for best multithreading always try to implement algorithms which require no state between frames. Whenever this is violated, be sure to group reads and writes to the state (do not spread them), and guard them in critical sections (as few and as short as possible). For example, it is a good practice to copy all your writable class variables (in a critical section) at the start of each frame into local stack variables, compute the whole frame outside of the critical section (updating the local variables that captured the global state as needed), then write them back together at the end of your frame in another critical section. Do not request an automatic lock around your whole filter from Avisynth, because it will serialize your filter's execution.
- If you have class variables that must be writable in every frame, you will also have to keep in mind that Avisynth does not guarantee that frames will be processed in their natural order. Just a reminder.
- Only store variables in classes and in method stacks. Per-frame heap allocations should be avoided, because they can act as implicit synchronization points between threads. And most importantly, never store anything in static variables or in the (global) namespace scope. Read the previous sentence a few more times.
- Do not store the IScriptEnvironment pointer anywhere yourself (except locally on the stack), and never reuse those pointers outside of the methods where they were supplied to you. Not even between different executions of the same method! There is a reason why you get that pointer separately for each method, which is that it may be different every time, especially in multithreaded scenarios. If you reuse it, the consequences will be different between every implementation, but you can get anything from race conditions to program crashes.

[Robert Martens]

Do not store the IScriptEnvironment pointer anywhere yourself (except locally on the stack), and never reuse those pointers outside of the methods where they were supplied to you. Not even between different executions of the same method! There is a reason why you get that pointer separately for each method, which is that it may be different every time, especially in multithreaded scenarios. If you reuse it, the consequences will be different between every implementation, but you can get anything from race conditions to program crashes.

I have to own up to being a dummy here and having done that, thinking the rule only applied to storing it as a global variable. I had made the--admittedly careless--assumption that storing the environment pointer as a non-static class variable would be safe since each instance of the host would have its own instance of my filter class. I take it I'm supposed to understand that multiple instances of a host application might try to access a single instance of my filter?

Only the slightest bit more consideration put into my problem and I've eliminated what I thought was the need to store the pointer, so it's no big deal for me to bring my code into compliance (luckily I work so slowly I haven't released a version with this problem included), but for the sake of curiosity I thought I'd ask.

[Myrsloik]

I have to own up to being a dummy here and having done that, thinking the rule only applied to storing it as a global variable. I had made the--admittedly careless--assumption that storing the environment pointer as a non-static class variable would be safe since each instance of the host would have its own instance of my filter class. I take it I'm supposed to understand that multiple instances of a host application might try to access a single instance of my filter?

Only the slightest bit more consideration put into my problem and I've eliminated what I thought was the need to store the pointer, so it's no big deal for me to bring my code into compliance (luckily I work so slowly I haven't released a version with this problem included), but for the sake of curiosity I thought I'd ask.

Always pass the environment pointer as an argument to your functions. It's a simple rule. Because Avisynth isn't multithreaded you'll never have two threads accessing your filter at once. However it's still a very bad practice. (not sure exactly what the MT side does to get around it)

Note that this issue affects everyone trying to use Avisynth plugins. For example RemoveGrain commits the biggest sin of all. It stores the environment pointer as a global variable when AvisynthPluginInit2 is called. This makes VapourSynth, which creates fake Avisynth environments only when needed, crash. A lot. Since the environment pointer will be different between plugin instances and registering functions.

I'd like to add another filter writing tip myself.

ONLY call AddFunction in AvisynthPluginInit2. Do not throw errors. DON'T DO ANYTHING ELSE THERE. Do initialization when one of your functions is called the first time.

It's the biggest reason for autoloading "breaking" in avisynth and a pile of other issues.

[ultim]

I take it I'm supposed to understand that multiple instances of a host application might try to access a single instance of my filter?

Almost. Only one host application will use your filter instance, but that single host application (the Avisynth library to be precise) can use different IScriptEnvironment objects depending on which thread your filter instance is running on. Avisynth-MT, for example, uses special IScriptEnvironments instances to implement thread-local storage (TLS), and I'm implementing something similar in Avs+ too. Apparently, VapourSynth makes use of fake IScriptEnvironments objects too. The thing about TLS objects are, well, that they are local to the thread, so if the given multithreading implementation (for example Avs-MT mode 1) creates just one filter instance for all threads, your filter will get different IScriptEnvironment pointers based on the thread it is running on. Because you cannot know in your filter what threading implementation of Avisynth is in use, it is best not to store the environment pointer between calls at all, and only then will you be safe in every case.

Because Avisynth isn't multithreaded you'll never have two threads accessing your filter at once.
Nonononono, forget what Myrsloik said here. This statement can be very misleading, because it is only true for some versions of Avisynth (like classic non-threaded version). The point of this discussion is to let plugins prepare for multithreaded versions (and/or multithreading modes). An important point of my tips is that if you can, you SHOULD assume that multiple threads will be accessing your filter at once, as it really can happen, even with existing implementations.

... in AvisynthPluginInit2. Do not throw errors.
A little bit off-topic (as far as multithreading issues go), but for best practices about exceptions, look at this tip (http://forum.doom9.org/showthread.php?p=1647262#post1647262).

[Robert Martens]

Thanks for the detail, that clears everything up! I've already eliminated the broken code and pushed the fix, now I just need to sort out one spot in my test executable where I'm catching an AvisynthError to grab its error message and I should be done with your list of Don'ts for the time being.

[Stereodude]

While I don't have anything technical to add, I would like to thank ultim and others for their efforts to modernize AVIsynth by dragging it kicking and screaming into this decade.

[ajp_anton]

If there's room for feature requests, can we skip some arguments when calling functions like this:
some_function(a,b,,d,e)
if I don't want to change the default value for the third argument, and don't want to explicitly use the names of the 4th and 5th arguments.

[Gavino]

If there's room for feature requests, can we skip some arguments when calling functions like this:
some_function(a,b,,d,e)
You can get that effect now (more or less) by using (from v2.60):
some_function(a,b,Undefined(),d,e)
or if you prefer
global _ = Undefined()
...
some_function(a,b,_,d,e)

I don't like the idea of allowing some_function(a,b,,d,e) because a user could easily put an extra comma by mistake and not realise he wasn't getting what he expected.

[l33tmeatwad]

Came across something interesting when testing things....if you have fft3dfilter (version 2.1.1) in a script twice and the bw & bh settings in one is 3x or more than the other then there is a green and red bar at the top. This odd glitch only happens in AviSynth+ and does not show up when using AviSynth 2.6 a5. Figured this was worth mentioning, I haven't tested this on another machine yet.

[cwk]

I also get the green bar when using the fft3dfilter, although I see it with just a single placement of the tool within a script. I am working around it for the time being by merging chroma back in at the end. I'll admit this means you can't use fft to filter your chroma.

[TurboPascal7]

Please post the minimal script you can reproduce the issue with (I couldn't). Also, what colorspace and what resolution you're working with?

[cwk]

I can see it with the following script


avisource("1_fields.avi")
converttoyv12()
fft3dfilter()


The source file is huffyuv, 688 x 476. I also see the lines if I resize to 704x480 or 720x480. fps is 59.940, if that matters.

[TurboPascal7]

Reproduced, thank you.
----
Okay, it's a bug in env->BitBlt.

fft3d calls it in CoverbufToPlanarPlane

for (h=0; h<dst_height; h++)
{
env->BitBlt(dstp, dst_pitch, coverbuf1, coverpitch, dst_width, 1); // copy pure frame size only
dstp += dst_pitch;
coverbuf1 += coverpitch;
}
That in BitBlt, passing the height == 1 condition, goes to
#ifdef X86_32
if (GetCPUFlags() & CPUF_INTEGER_SSE)
memcpy_amd(dstp, srcp, src_pitch*height); // SSE
else
#endif
memcpy(dstp, srcp, src_pitch*height); // fallback
}
which in any case ends up broken because you actually have to copy row_size bytes instead of src_pitch bytes in this case.

So either we could remove the height == 1 check, replace src_pitch with row_size (they're equal when height != 1) or fix fft3d to call BitBlt once for the whole frame instead of every line.

Original Avisynth actually does it right:
if (height == 1 || (src_pitch == dst_pitch && dst_pitch == row_size)) {
memcpy_amd(dstp, srcp, row_size*height);
} else {
asm_BitBlt_ISSE(dstp,dst_pitch,srcp,src_pitch,row_size,height);
}

[ultim]

Okay, it's a bug in env->BitBlt.


You might be right. Actually I'm greatly annoyed by Avisynth's BitBlt, IMHO it is a messed up implementation, but when I "fixed" it last time, it broke some plugins doing interleaved copy of non-interleaved frames. Even today I still don't know if the ability to do such copies is intentional, or is it an accident that plugins started to rely upon. Anyway, I changed "row_size*height" to "src_pitch*height" because I thought it is much more logical, thinking they are the same anyway because of the condition row_size==dst_pitch==src_pitch that guards it. Here is where I made the error: I accidently overlooked that this last condition need not be true if height==1. Fix in next release. And thx for looking into this for me.

edit: added clarification of what plugins got broken.

[Guest]

"row_size==dst_pitch==src_pitch"

Are you sure about that? I was under the impression that a Crop can change that. It's always been a common error with Avisynth plugins to assume that they are the same.

[TurboPascal7]

"row_size==dst_pitch==src_pitch"

Are you sure about that? I was under the impression that a Crop can change that. It's always been a common error with Avisynth plugins to assume that they are the same.
This is irrelevant. It's simply a condition (https://github.com/pylorak/avisynth/blob/master/avs_core/core/bitblt.cpp#L269). If it isn't true - the code will go the other way, that is all.

[Guest]

Thanks for the lesson on coding but as a condition "row_size==dst_pitch==src_pitch" is nonsense.

[ultim]

"row_size==dst_pitch==src_pitch"

Are you sure about that? I was under the impression that a Crop can change that. It's always been a common error with Avisynth plugins to assume that they are the same.

Of course that equality needs not be true, not just because of crop, also because of alignment. But the piece of code in question here is guarded by an "if" to be able to take a more optimal path. It does not actually require that row_size==dst_pitch==src_pitch, if that is unmet, it will take a slower path.

Edit: I do find it a nonsense that the code is therefor optimized for the minority of the cases, which is (only) part of the reason I hate the BitBlt implementation. Fixing that though is not trivial due to existing plugins relying on the current implementation.

[TurboPascal7]

neuron2, no problems, don't hesitate to ask. By the way, this condition is also present in the original avs core (https://github.com/jeeb/avisynth/blob/master/src/core/avisynth.cpp#L1904). So you're a bit late with the nonsense part.

[Guest]

LOL. I don't actually need any help or guidance from you in SW design or coding. The stated condition was nonsense in the post that I responded to. Let it go, friend, or take it to PM.

[ultim]

If there's room for feature requests, can we skip some arguments when calling functions like this:
some_function(a,b,,d,e)
if I don't want to change the default value for the third argument, and don't want to explicitly use the names of the 4th and 5th arguments.

Sure, there is always room for feature requests! So please post everything on your mind here on the forums, or as a GitHub issue. It allows us to discuss your ideas. In this particular case, I agree with Gavino, that the syntax you demonstrated would let many typos go undiscovered, but Gavino proposed a nice way to get what you'd like without its dangers. Cheers to both of you.

[raffriff42]

Sure, there is always room for feature requests!How about this syntax change...for better self-documenting function declarations:

(EDIT or would this be out because it breaks compatibility?)## function declare examples, current syntax:

function SAA(clip C, int "ss", bool "cp", int "aa")

function FunkyDeblock(clip c, int "quant", int "th",
\ int "radius", bool "deblock", bool "depump")

function Detwinkle(clip C, float "grainmask",
\ int "grainmode", int "tstren",
\ float "hicont", float "hisharp",
\ float "unblur", float "mosens",
\ int "hismooth", String "show")

## add syntax support for DEFAULT VALUE:

## * defaults are optional of course
## * compiler assigns default values automatically if specified
## * string literals only - no variables or Eval statements!
## * strings are not quoted (certain chars can't be supported - unless escaped?)
## * empty String defaults accepted, meaning empty string, as opposed to Undefined.
## * other empty defaults are syntax errors
## * specify explicit undefined defaults with "Undefined", "Nop" or "null"

function SAA(clip C, int "ss=2", bool "cp=true", int "aa=48")

function FunkyDeblock(clip c, int "quant=4", int "th=3",
\ int "radius=4", bool "deblock=true", bool "depump=true")

function Detwinkle(clip C, float "grainmask=1.3",
\ int "grainmode=2", int "tstren=25",
\ float "hicont=1.3", float "hisharp=0",
\ float "unblur=3", float "mosens=3",
\ int "hismooth=2", String "show=")


## add syntax support for DEFAULT VALUE and VALID INPUTS:

## * compiler assigns default values and enforces valid inputs
## * space optional before start of "[...]" range specifier
## * ranges ".." delimited? Dunno.
## * valid alternate values "|" delimited
## * valid ranges: "[min..max]" "[min..]" "[..max]" or "[alt1|alt2|alt3]"
## * consider enforcing range specifier if default is used

function SAA(clip C, int "ss=2 [1..8]", bool "cp=true", int "aa=48 [4..256]")

function FunkyDeblock(clip c, int "quant=4 [0..51]", int "th=3 [0..128]",
\ int "radius=4 [1..60]", bool "deblock=true", bool "depump=true")

function Detwinkle[clip C, float "grainmask=1.3 [0.7..18]",
\ int "grainmode=2 [0..18]", int "tstren=25 [1..255]",
\ float "hicont=1.3 [0.1..10]", float "hisharp=0 [-1..1]",
\ float "unblur=3 [0..99]", float "mosens=3 [0.1..5]",
\ int "hismooth=2 [1..5]", String "show= [blur|mask|luma|cyan|]")

[ultim]

(EDIT or would this be out because it breaks compatibility?)

That's not a problem. As long as old Avisynth scripts can continue to run on Avs+, it is fine. I personally don't require that Avs+ scripts must also run on Avisynth, because otherwise a lot of improvements wouldn't be possible.

As for the feature at hand, I like the ideas, though I'd rather have some syntax details different. For example, the specification of the default values as per you wouldn't let the complier know the value's type. Only putting quotes around the var's name solves this, or in case a default values is supplied, not putting any quotes at all, because parameters that have a default are optional by definition.

[ultim]

Finally, some time to write down things on my mind. Well, not all the things, but it's a start ;)

Everybody hates MT-modes, me included. They shouldn't be required, but unfortunately we cannot dispose of them, because there are a lot of old plugins out there that were written without any consideration for multithreading. Not supporting MT modes would mean throwing out a lot of great plugins, keeping only those that perfectly adhere to the points in my previous "Plugin Writing Tips" post (http://forum.doom9.org/showthread.php?p=1649886#post1649886). So I've been wondering, what MT-modes should be supported? We know Avisynth-MT supports at least 5, but which of those are really usefull? I have recently recognized that the Avisynth-MT OP post has been updated, and now it only lists modes 1-3 as "usefull". I've set out to compile my own list, while in the process answering questions such as "When exactly do we need those modes?", "What modes are needed if no plugins are fixed?", "What modes are needed if all plugins are fixed (as per my plugin writing tips)?" and also "Which plugins cannot be used with the recommended set of MT-modes (1-3) of Avisynth-MT"?

I started to think analytically, and compiled a set of three tables found here (https://docs.google.com/spreadsheet/ccc?key=0AqVT4Ywc6rzYdHNoSm9mUWVDMWM4YTJvTTdEZFNlN1E&usp=sharing) to summarize my findings. I might have overlooked some things or made a mistake or two, so let me know if you find anything. Each table lists four negative plugin properties as their columns, basically violations against multithreading support. The last column contains the protection that Avisynth+ would need to implement to keep that kind of plugin working correctly despite its violations.

Table 1. has all the different combinations of properties, irrespective of if that combination is even realistic at all or not. For example, there can be no plugin such as its 9. row, because a plugin can only possibly require ordered frame retrieval if it stores at least some kind of internal state. There are also cases of inherently broken plugins that cannot be guarded against in Avisynth, because they'd require limiting the plugin simultaneously to both single AND to multiple instances. So, let's see, how many MT-modes do we need? Six MT-modes, ouch. More precisely, six MT-modes if we do want to support all possible kinds of "broken" plugins, without fixing any of them. This is where I've said "no way I'm gonna implement all those MT-modes"! So what can we do? Fix the plugins of course!

One of the easiest plugin fixes is getting rid of storing the environment pointer in classes or globals. Fixing these violations has some nice advantages: First of all, this is a trivial fix in any plugin, so as long as we have the sources, this kind of fixing will go fast and easy. Also, quite importantly, getting rid of this single violation in every plugin will effectively ensure that there is at least a way to use all plugins in multithreaded modes (as long as the required protections are implemented in the core). So, never store the env.ptr. in classes and globals, and we are down to only 7 cases and 4 MT-modes (3 protections). Welcome to table 2. The protections in table 2. are what we'd need to have if we could perform only a trivial fix in the few plugins that need it.

Getting to table 3. from here is actually quite simple, only two simple facts have been used for this transformation. One is to get rid of using global variables. This is also a simple fix, as basically all that needs to be done, is to make class variables out of global variables. This will not effect plugin behavior at all, but it will remove the need to serialize these plugins, which is practically the same as letting them run fully parallel. Believe me, this fix is almost just copy-paste work for many plugins (not for those whose algorithm requires it, but those can be fixed too in every case, except that more work needs to be done). The main obstacle will (again) be having the sources to the offending plugins. OK, so far so good. But to get to table 3., I've also done something else: I've assumed that there are no plugins that must serially access all frames up to frame n, to calculate frame n. Why? Because such a plugin would hardly be usable in Avisynth (as it would make it impossible to seek inside or trim the video), and thus I actually doubt that one like that exists at all. (EDIT: I've been let to know that there are such plugins in existence. But all Avs+ could do to protect them is to play back all frames of the video internally until the requested frame is reached, which is performance-wise not practical at all - think of having to wait for multiple hours just to seek into the middle of the video clip).

So, we're at table 3., and IMHO have don't need any complex plugin fixes. Table 3. summarizes the MT modes that we need to implement if we assume we can fix all plugins until at least none of them uses global vars. Is that realistic? Yes, as long as we have the sources. Surprisingly to me, the conclusion in this table is that we either need no protection at all, or a single protection for unfixed plugins that store state across frames, but that still allows for full performance/parallelization, only with memory overhead (fully fixed plugins never require any protection from Avisynth). A second protection (3rd mode) is added to optimize source filters further, and while tbh I don't think it is strictly needed for correct source filters, it should still help their performance by reducing disk seeks. Another (important but sad) advantage of having this special mode for source filters, is that the same protection also lets us use unfixed plugins that use global or static variables. Note that while this special mode allows you to use such plugins, you should not rely on this mode (except for source filters), because it will cripple your multicore performance and flush it down the toilet, instead it is better to work on the plugin to not require a single serialized instance.

Avisynth+ will likely implement the modes listed in table 3., and I will take time to try and fix/modify as many plugins as possible, as well as motivate others to do so. In contrast to Avisynth-MT though, Avisynth+ will also allow plugins to register themself for the correct MT-mode they need, so users won't have to know by heart for all plugins if they have been fixed, or if they are unfixed, which mode they need.


Congrats for reading this rant of mine. What's to take home ?
- Fix plugins that store and reuse the env. ptr. It is often neglected but very important.
- Fix plugins that use global or static variables. We can guard against them to ensure they will work correctly, but they cannot be executed in parallel, and will seriously limit the multithreaded performance for your whole filterchain (even for correct plugins, due to frame dependencies).
- You can violate multiple rules for multithreading, and not all are equally important. Most importantly, ensure that the env.ptr. is never stored and reused. If you still have energy left after that, fix the plugin further so that no global/static state is used. Then if you still have energy left, try to turn all class members into read-only.
- The three MT-modes listed as "usefull" for Avisynth-MT really are the three most usefull MT-modes.
- The MT-modes of Avisynth-MT are not enough to use every single plugin when MT-mode is enabled. You might get crashes or corrupted frames if you use an unsupported plugin in an MT-mode, but don't blame Avisynth-MT, it is really not his fault. Blame the author of the offending plugin.
- Always fix plugins instead of relying on MT-protections. Most fixes are very easy and you will get much better performance than using a built-in protection.
- If you have the sources to a plugin, please don't keep it to yourself.

[TurboPascal7]

Okay, here are my thoughts on the issue. Everything runs on the assumption that most filters will get updated. There are just too many things to add and I want to hope that with the help of VSynth team we will be able to make the most used plugins less bad quite fast.

Common things:

In any case the core should include a static thread pool with number of threads configured on script initialization. Additional function like SetThreads(4) should be provided with all calls other than the first one ignored. This is exactly how Avstp works.
Nothing should be threaded by default. If the core does not have any information on the filter it will run singlethreaded. Always.

The question is: how does the core get the required info if the filter does not register itself?

1. Single-line SetMtMode
First option is to make the user put this info in his script but unlike the original SetMtMode apply threading only to the filter inside the call. The result script would look like:
SetThreads(4)
Source(...)
SomeRegisteredFilter() # registers itself as threaded, runs with 4 threads
SomeBrokenFilter() # does not register itself, singlethreaded
ForceThreading(SomeFilter(), 2) # SomeFilter runs in mode 2, forced threading

This assumes that avisynth evaluator would allow making SomeFilter() threaded and you don't have to use like SetMtMode(2).SomeFilter().SetMtMode(0) which is quite ugly.
Compared to original mt-mode solutions, this makes immediately obvious where and which mt-mode is used. Of course it runs on assumption that most filers will get updated, otherwise you'll end up writing ForceThreading call on every line or separating all your stuff into functions just to thread said function in one call.

However, just like the vanilla SetMtMode solution, this has some serious disadvantages.
First of all, it requires users to know implementation details of every plugin and script they're using, in other words - making the user learn a lot of stuff he doesn't care about.
Second, it makes your script less readable by simply adding garbage calls to it.
Third, imagine you're using some dither.avsi function with 50 calls to perfectly threadable masktools and a single call to hopelessly broken removegrain. You'll end up using the slowest mt mode of all (if any) unless you're willing to modify this script. No one will do that.

So I think we should face it: SetMtMode solution is terrible. I'd rather continue running scripts singlethreaded than spending hours and hours understanding what I can and cannot thread just to later end up with a broken encode because I used a wrong mt mode.

2. Embed this information into the core
Do pretty much the same thing VSynth is doing (https://github.com/vapoursynth/vapoursynth/blob/master/src/avisynth/avisynth_compat.cpp#L207): embed all the required info into the core. Here is how it'd work:

The core has predefined dictionary of mt modes for every plugin known. {"mt_lut": 0, "removegrain": 2} and so on. User has no ways of providing this info. No SetMtMode function exists.
When a threaded plugin is loaded, its registration info overrides the default dictionary values. This is important because we'll never be able to keep the list up to date.
All filters that do not register themselves and don't have an entry in the core dictionary run singlethreaded.
When a filter gets updated, we simply remove its entry from the core. This makes it singlethreaded for users that updated the core but didn't update the filter. Can't be helped.

Yes, this solution is harder for the devs since one would need to gather the info required to embed it in the core. Yes, this solution is not optimal from performance point of view - even if the filter can be easily threaded, unless it registers itself or has an entry in the core dictionary - it won't.
But it does not require the user to know anything about filters he's using. Plus it doesn't add anything to your scripts. Plus you automatically get threading even inside complicated scripts that no one will ever modify. User updates avisynth.dll and it just works. Then when plugins get updated, he replaced the files in his plugins folder and it just works.

I definitely think this is the way to go. MAXIMUM PERFORMANCE is cool and everything, but I really want things to just work without me doing anything. In the end, 20 minutes of my time is much more important for me than those four additional hours my PC will spend on filtering running some more stuff singlethreaded.

[ajp_anton]

Bug?
subtitle doesn't accept an argument named "text". The text has to be the first unnamed argument for it to work. I was trying to re-organize the arguments when calling subtitle for easier readability.

Feature request: allow comments on split lines, like this:
filter(argument 1 #comment
\, argument 2)

[ultim]

Bug?
subtitle doesn't accept an argument named "text". The text has to be the first unnamed argument for it to work. I was trying to re-organize the arguments when calling subtitle for easier readability.
No, not a bug. The "text" argument is mandatory, meaning it is not a named argument. Because it is not a named argument, you cannot reorder it and refer to it by name.


Feature request: allow comments on split lines, like this:
filter(argument 1 #comment
\, argument 2)
Noted. Thx.

[Gavino]

Feature request: allow comments on split lines, like this:
filter(argument 1 #comment
\, argument 2)
You can use the /* ... */ style of comment to do this (in both Avisynth and Avisynth+)
filter(argument 1 /* comment */
\, argument 2)

[LaTo]

Surprisingly to me, the conclusion in this table is that we either need no protection at all, or a single protection for unfixed plugins that store state across frames, but that still allows for full performance/parallelization, only with memory overhead (fully fixed plugins never require any protection from Avisynth).

If we really need some writable class member, what is a good method?

(I was thinking about a wrapper which dispatch data between threads using a hashmap of thread id)

[TurboPascal7]

If we really need some writable class member, what is a good method?

(I was thinking about a wrapper which dispatch data between threads using a hashmap of thread id)
There is also an option of letting the core create a few instances of your filter class. This way you won't have to implement any kind of synchronization or complicate your plugin.

Using a hashmap in this case is quite hard. You will either have to implement a thread-safe hashmap or know all possible filter ids during hashmap creation in filter constructor. First way will complicate your implementation a lot, the other one will bind your plugin to avs+, plus it is somewhat a bad api design.

[ultim]

If we really need some writable class member, what is a good method?

(I was thinking about a wrapper which dispatch data between threads using a hashmap of thread id)

You don't need to do that kind of dispatching on your own.

The "best" (ahm) way depends on how you'll use the class member. If you don't mind missing out some frames here and there, the easiest way for you as a filter author is to not implement any synchronization, and let Avisynth create multiple instances of your filter. This will of course come with some overhead, mostly some memory, but it's the quick'n dirty method, because from your filter's point of view there will always be just one thread.

On the other hand, if you'd want to share your state between all your instances (and you don't want to miss out on any of the frames), then you'll want to implement your own synchronization, and ask for the 1st MT mode where Avisynth does not provide any "workaround"/protection. In this case Avs+ will create just a single instance of your filter (which will be the same one called from every thread), and you'll need to synchronize access to the state yourself. Note though that from your point of view, the frames might (probably) still not be accessed in sequential order.

So to sum up, the first method is useful if you only need to store some temporal state, but you don't mind if your filter's instance will not see some frames. Otherwise, use the second method. Jugding by the fact that you wanted to dispatch data based on the thread ID, you probably want to use the solution involving multiple instances.

(And for future reference, if you really do want to dispatch data yourself, you don't need to use a hashmap. Avs+ will sequentially number the threads for you, so you'll be able to use a simple array lookup. You'll tie your plugin to Avs+ by that though.)

[SEt]

Don't bother with MT modes if you want future-proof solution. It's a ok hack for doing work right now, but it's not scalable. I have much better idea how efficient threading should be done, but never got to implement it (mostly because I'm not pressed since MT modes do work for now).

That have been said, I'm busy with other areas atm. (And you will likely see something useful for Avisynth in several days. ;))

[TurboPascal7]

I have much better idea how efficient threading should be done
You could tell us about this idea of yours instead of just saying you have it, you know.

[TurboPascal7]

Okay, this projects needs some user input again.

A new issue got created on github (https://github.com/pylorak/avisynth/issues/10) but unfortunately we couldn't decide whether it should be implemented or not. This is not a critical issue of any sort but it is quite annoying.

How to reproduce
The minimal example as posted on github is:
blankclip(1000,720,480, "yv12",color_yuv=$808080)
anotherfilter = blankclip(1000,720,480, "yv12",color_yuv=$FF8080)
StackHorizontal(last, anotherfilter)
It works fine. Now if you remove the last line, making the script
blankclip(1000,720,480, "yv12",color_yuv=$808080)
anotherfilter = blankclip(1000,720,480, "yv12",color_yuv=$FF8080)
you will get a "Not a clip" error.

Why
Assignment operator returns uninitialized avisynth value, basically nothing. This "nothing" does not override the last variable so you still can request that one simply putting last on the next line.

Proposed solution
Instead of returning nothing, return the last clip value if it exists. Same effect could be achieved right now by adding last to the line after every assignment in your script.

Drawbacks
I don't see how this could break any existing script. It might reduce performance of script initialization a bit since it requires looking up the last variable on every assignment, but you won't notice this. Of course I might be missing something.

Why bother
You write a script and then you decide to try it without a few lines at the end, so you comment them out and BAM - "not a clip". This could be solved by always adding "last" to the last line of your scripts but it's kinda hacky.

What's needed
Your input. Basically, something like "oh I had this problem before, would be nice to get this fixed" or "nah, this will break my scripts because I found a way to rely on this behavior", or even "omg no, this will reduce initialization performance by 0.00000001%, can't have that" would work.
IanB's opinion if we could fix this in the vanilla avisynth would be really appreciated.

Thanks in advance.

[wOxxOm]

It might reduce performance of script initialization
There won't be any perceivable penalty at all in case avisynth+ would implicitly add 'last' only to the end of a scope (function/eval snippet/entire script).

[Gavino]

If I understand correctly, the proposal is that a script that ends with an assignment should return 'last', instead of a 'void' value as currently. I assume the same would apply to functions ending with an assignment (and also to Eval() contents).

I don't expect any currently working scripts (or functions) would break as a result.
However, a script that ends with an assignment is probably wrong. Why assign to a variable that will never be used?
Who knows what the user intended in such a case?
Perhaps he meant to return the value being assigned, rather than a 'last' that may have been set many lines earlier.
So I think it is better to keep the existing scheme, which forces the user to make the intention clear.

There won't be any perceivable penalty at all in case avisynth+ would implicitly add 'last' only to the end of a scope (function/eval snippet/entire script).
However, it must only do this for assignments. With the existing data structures, at the time the end of the scope is being evaluated, you no longer know whether you had an assignment - and when evaluating an assignment, you don't know whether it's the last thing in its scope. So the simplest implementation, if going ahead with this, would be for all assignments to return 'last' as their value.

[innocenat]

I think implicitly return last variable on every script is a good idea. Sure, it might not seems resonable at first, but if you need temporary storage in your script you probably aren't beginner and should know what you are doing and what last variable refer to anyway.

As for multithreading, I dont think creating multiple instances of filter is a good idea. I think avs interfacd should provide filter thread local storage and thread-safe global storage. On filter that require linear frame access, filter should be registered as such, but avs interface should provide a way for filter to tell the core that the process that require linear access has done and the core is allowed to request next frame while this frame is still processing.

I know this is not a light work and may break compatibility, but it's how I imagine it right now.

[TurboPascal7]

However, a script that ends with an assignment is probably wrong. Why assign to a variable that will never be used?
Who knows what the user intended in such a case?
Perhaps he meant to return the value being assigned, rather than a 'last' that may have been set many lines earlier.
So I think it is better to keep the existing scheme, which forces the user to make the intention clear.
This is a job for an analysis tool and not for the core itself, imho. I don't see how assignment on the last line is wrong while having any number of unused variables in other parts of the script is okay.

As for multithreading, I dont think creating multiple instances of filter is a good idea. I think avs interfacd should provide filter thread local storage and thread-safe global storage.
There is a "small" problem with this solution - it ties the plugin to avs+. Maybe it would be okay if we were starting from scratch, but with o9k avisynth forks around I think it would be reasonable to avoid coupling the plugins to avs+ interface as much as possible. Not to mention that the core will need to implement the multiple class instances solution anyway just to thread some existing plugins without modifications.

[cwk]

Can we assume this only attempts to solve the case where the last variable received a valid video clip, just fails to return it?

Consider the case where a script is calling functions that are under construction.


mySource = avisource("blah")
a = mySource.SomeCompletelyAwesomeFunction()
return a


Here 'not a clip' is useful, because it tells me my function is not returning a valid clip.

Another approach would be to make the existing 'not a clip' message more informative. It could tell the user what the current return value is, or point out the line providing the return.

[TurboPascal7]

Can we assume this only attempts to solve the case where the last variable received a valid video clip, just fails to return it?
This proposal would change exactly nothing in your example if this is what you're asking about.

[Reel.Deel]

As a novice AviSynth user I had this problem a few times, and when someone is new to AVS this could be kind of a turn off. Now that I know better and I'm used to this behavior would I like to see a change? Not really, but a more comprehensive error message would definitely be a good idea.

@TurboPascal7
On a side note, any plans to integrate FTurn into AviSynth+?

[wOxxOm]

On the 'last':

a typical mix of filter calls and assignment statements:mainfilter1()
b = auxfilter()
# last = mainfilter1() as expected
mainfilter2()
-----
now someone commented out just the last line knowing how 'last' clip is usually preserved over assignment statements, thus expecting the 'last' clip to carry result of the last filter applied in - what should we call it? - the main flow of the filter graph.mainfilter1()
b = auxfilter()
# mainfilter2()last = nothing????

So, irregardless of user experience, there seems to be no logic *syntax-wise* behind current behavior (but of course if one looks into the source code it's clear that filters() explicitly try to use the 'last' clip in rather an unsightly manner, which doesn't compensate the lack of syntax logic anyway).

[TurboPascal7]

On a side note, any plans to integrate FTurn into AviSynth+?
Will be in the next version or the one after that.

[Daiz]

Not a clip

I've run into this so many times I've lost count, usually in the exact situation described (testing filtering and commenting out a line), so here's a very large "yes, please make assignment return last" from here.

[Groucho2004]

However, a script that ends with an assignment is probably wrong. Why assign to a variable that will never be used?
Who knows what the user intended in such a case?
Perhaps he meant to return the value being assigned, rather than a 'last' that may have been set many lines earlier.
So I think it is better to keep the existing scheme, which forces the user to make the intention clear.
Exactly my thoughts. Also, less experienced users would probably in many cases not know what the script returns with this modification.
When you write a function in C for example, the compiler forces you to return from that function with the correct type and that makes sense. You don't dumb down the language because of convenience.

[TurboPascal7]

Exactly my thoughts. Also, less experienced users would probably in many cases not know what the script returns with this modification.
When you write a function in C for example, the compiler forces you to return from that function with the correct type and that makes sense. You don't dumb down the language because of convenience.
Avisynth is not C. Stop applying rules of normal languages to it. You don't want to write "return derp" in the end of every function/script, do you?

As for assignment on the last line being wrong argument, what about scripts like this:

super=MSuper(last, pel=4, planar=true)
pp_super=fft3dgpu().MSuper(pel=4, planar=true)

b1vec = MAnalyse(super, delta=1, isb=true, blksize=8, overlap=4,search=5)
f1vec = MAnalyse(super, delta=1, blksize=8, overlap=4,search=5)

b1clip = MCompensate(orig, super, b1vec, thSAD=400)
f1clip = MCompensate(orig, super, f1vec, thSAD=400)
Whoops, variable pp_super is unused, the user ended up calculating vectors on the wrong clip. This is a bug. And we did exactly nothing to prevent it. Now please explain me the reason why we should do something to prevent this kind of behavior if this assignment happens on the last line. Because it's easier to implement? Because it's always been like this? I think we either go full way preventing users from shooting in their own feet and forbid unused variables altogether (this is a bad idea please don't even consider this) or drop this kind of argument.

As for users not knowing what script returns: if this behavior got silently changed, how many people do you think would notice? :)

P.S. I'm sorry if I sound too "rude" and I should probably stop posting on the issue. The question was posted to get opinions, not arguing about them. Sorry.

[Groucho2004]

Avisynth is not C.
Thank you for your wise words. I was using it as an example, not to be taken literally. The logic still applies. Your logic is just different.
You don't want to write "return derp" in the end of every function/script, do you?
I kinda do...

[Sapo84]

Whoops, variable pp_super is unused, the user ended up calculating vectors on the wrong clip. This is a bug.

It depends, maybe the user was using pp_super before, didn't like the result and stopped using it without removing the variable.

Anyway, I'm against a fix that will allow sloppy coding (commenting lines and leaving an assignment as the last line sounds like sloppy coding to me), instead of commenting just use a return last in the place you need to test the output.

[Gavino]

Now please explain me the reason why we should do something to prevent this kind of behavior if this assignment happens on the last line.
Because in that the case the problem is more than just the unused variable (that's just an aspect I mentioned to illustrate that the script was incomplete). The issue is that the intended return value is not clear.

instead of commenting just use a return last in the place you need to test the output.
Yes, I was about to make that point myself.

[wOxxOm]

Commenting was just an example, there's no need to focus on that per se.

The real problem is the quirky behavior of an assignment statement in avisynth at the end of a scope.
It is inconsistent.

Either make all assignment statements kill the 'last' clip (nonsense), or make the last assignment behave exactly the same as the others.

[TurboPascal7]

Because in that the case the problem is more than just the unused variable (that's just an aspect I mentioned to illustrate that the script was incomplete). The issue is that the intended return value is not clear.
function test() {
colorbars(640, 480)
clp = blur(1)
}

colorbars(320, 240)
test()
blur(1.0)

What did the user intend to return here? This script actually works, with function basically being a nop() call. Talk about preventing users errors. =)

Anyway, this was actually pointed out as an example where this proposal would break stuff, which for me is a showstopper. So I'm gonna quit here. But I (and not only I) would still love to see your ideas on multithreading. ;)

[Gavino]

make the last assignment behave exactly the same as the others.
It does behave the same as the others, there is no inconsistency.

Note that the commonly held belief that 'last' is returned by default is wrong. See this post.

[wOxxOm]

It does behave the same as the others, there is no inconsistency.Internally, yes. But not from a user's point of view.
Note that the commonly held belief that 'last' is returned by default is wrong.It's 'wrong' only because of the issue we're talking about.

[Gavino]

Internally, yes. But not from a user's point of view.
What is the inconsistency from the user's point of view?

It's 'wrong' only because of the issue we're talking about.
Not just this issue (assignment) - if the last thing is a non-clip expression (eg an int), 'last' is not returned either, even if set earlier.

The Fluff had an excellent post just now describing how it works in detail, but it seems to have disappeared. :(

[wOxxOm]

What is the inconsistency from the user's point of view?Well, I thought it's shown in my posts above...
Not just this issue (assignment) - if the last thing is a non-clip expression (eg an int), 'last' is not returned either, even if set earlier.If it's not an assignment then from user's point of view it's the same as any other filter expression, so the 'last' is returned as expected.

[TheFluff]

I deleted the post because I thought the discussion had already covered what I was talking about but here's the relevant part again:

Avisynth has four types of expressions:
1) explicit returns ("return <expression>")
2) explicit assignments ("variable = <expression>")
3) implicit assignments to last (all expressions that aren't one of the two first types and evaluate to a clip)
4) other expressions where the return value is thrown away (everything else)

When the last expression in a block (either a function or a script) is not an explicit return (type 1 above), Avisynth will evaluate the expression and implicitly return the value it evaluates to, regardless of what the type of that value is (clip, int, string, undefined, whatever). The quirky thing here is that while expressions of type 3 and 4 above return the value of the expression, an explicit assignment returns an undefined value. I think this is a lazy fix to make sure an explicit assignment of a clip to an arbitrary variable doesn't overwrite last, which I suspect would happen if the assignment returned the assigned value like every other assignment operator on the planet does. The inconsistency is that an explicit assignment returns undefined while an implicit assignment to last returns the assigned value.

What TurboPascal7 proposed to change is simply the implicit return mechanism: if you attempt to implicitly return an explicit assignment expression, last should be returned instead of the undefined value, if last is defined at that point. The problem with this is that there's nothing about the scripting language that forces you to use last, and thus there's really nothing that says that at the end of a block the contents of last would be a better or more correct value to return than any other arbitrary value. I can accept an operator returning an undefined value, even if I think that's ugly, but having an operator return a defined, arbitrary value that may be completely unrelated to any of its operands is beyond ugly.

What I think you should do is promote the assignment operator to full citizen status like all of Avisynth's other operators, give it a place in the precedence table and make it return the assigned value like assignment operators in basically every other language do. Then fix the script parser so this doesn't overwrite last when the assigned value is a clip. This will probably break things though.

[ultim]

Yesterday we discussed this issue extensively with a couple of folks on IRC, and because there was no consensus, I encouraged them to bring this to the forums, to see what others think. I take Gavino's, Fluff's, Sapo's and Groucho's side here, and their arguments are the same why I wouldn't like to make this modification. To reiterate my points (which practically already have been said here by others), IMHO integrating this fix is not a good idea because:
- It will lead to confusion with non-expert users. If they expect anything to be returned after an assignment, they will expect the assigned value to be returned, not some hidden variable ('last') from a couple of lines earlier.
- More importantly, it can (and probably will) lead to hard-to-discover bugs, even with more experienced users. If the last statement is an assignment, did the user want the assigned value to be returned, or did he forget a line with 'last'? Changing the behavior to which basically equals to adding an implicit 'last' statement will hide one of these errors, while the current behavior makes the user spell out his expectations exactly (which avoids both error possibilities).

Wishful thinking on my part, but if we really wanted to make return values more logical and easier to use, we should either (1) return the last assigned value whatever it is even if not 'last', or (2, preferred) get rid of implicit returns completely and always require a 'return' statement in any block for a variable to be returned. Of course, both of these solutions would seriously break a lot of scripts, so obviously none of these two are gonna happen.

[Gavino]

I don't want to prolong the debate unnecessarily, as it seems we have now reached a conclusion, but just to address a couple of points raised...

If it's not an assignment then from user's point of view it's the same as any other filter expression, so the 'last' is returned as expected.
Consider
function f(clip c) {
c
...
width()
}
Here, we don't have an assignment but an int expression.
The value of that expression is returned rather than the value of 'last' (which is 'c').
(Note that only clip values are ever implicitly assigned to 'last'.)

The inconsistency is that an explicit assignment returns undefined while an implicit assignment to last returns the assigned value.
Strictly speaking, there is no implicit assignment to 'last' on the last statement. Instead, it is treated as an implicit 'return' (and defined this way in the docs).
If you look at it this way, then there is no inconsistency in the treatment of assignments.

[Lenchik]

"nah, this will break my scripts because I found a way to rely on this behavior" or something like that. I got used to current behaviour. And moreover i am often creating chains of scripts by importing them into one biggest. I think that new modification can make more difficulties than making life easier.

[SEt]

You could tell us about this idea of yours instead of just saying you have it, you know.
I doubt anyone it going to implement it, but as you requested, general outline:


In one word I can describe it as "superscalar" architecture. The same ideas as used in modern superscalar CPUs:

1) Work is separated from data flow. (-> denotes function call)
How it goes now: Render->FilterB->FilterA->Source.
How it should be: Scheduler->Source, Scheduler->FilterA, Scheduler->FilterB.
2) Filter has to be able to tell the scheduler which frame numbers from which its sources it needs to produce each destination frame number. Fast, without actual frames, but maybe imprecise (can fail its processing call and request more).
3) There are several kinds of filters: one instance multicall (like MTMode 1), one instance singlecall (MTMode 3), multi instance one call per instance (MTMode 2). Type is provided by the filter itself, always.
4) Based on 1-3, number of available hardware threads and data flow graph Scheduler builds filter call plan in superscalar way: several frames in flight on different stages.
5) Refcounting and no cache. Based on call plan each frame has its own reference count and will be kept exactly until all references are used. All produced frames are read-only, but could be forwarded (for filters like Trim).

As could be seen, such architecture not only would allow efficient and scalable utilization of hardware threads but will be able to incorporate even extremely threading-unfriendly filters with sequential frame processing without any correctness or speed drawbacks given enough other work in script.

Also it would be very memory-efficient: only actually needed data is kept and memory is consumed as much as needed for no redundant work in the scheduling window.

Cons: hard to implement, scheduling is active and will consume some resources, how filters work needs to be modified: source frames are provided to filter, not requested from inside.


Legacy Avisynth 2.5 filters work through separate filter "wrapper". Threading information is provided to it in textual user-editable file config.

[ultim]

"nah, this will break my scripts because I found a way to rely on this behavior" or something like that. I got used to current behaviour. And moreover i am often creating chains of scripts by importing them into one biggest. I think that new modification can make more difficulties than making life easier.

actually, i'm pretty confident that the change wouldn't break any existing scripts. from that point of view it should be totally harmless. i have other concerns, see my previous post.

[ultim]

I doubt anyone it going to implement it, but as you requested, general outline:


In one word I can describe it as "superscalar" architecture. The same ideas as used in modern superscalar CPUs:

1) Work is separated from data flow. (-> denotes function call)
How it goes now: Render->FilterB->FilterA->Source.
How it should be: Scheduler->Source, Scheduler->FilterA, Scheduler->FilterB.
2) Filter has to be able to tell the scheduler which frame numbers from which its sources it needs to produce each destination frame number. Fast, without actual frames, but maybe imprecise (can fail its processing call and request more).
3) There are several kinds of filters: one instance multicall (like MTMode 1), one instance singlecall (MTMode 3), multi instance one call per instance (MTMode 2). Type is provided by the filter itself, always.
4) Based on 1-3, number of available hardware threads and data flow graph Scheduler builds filter call plan in superscalar way: several frames in flight on different stages.
5) Refcounting and no cache. Based on call plan each frame has its own reference count and will be kept exactly until all references are used. All produced frames are read-only, but could be forwarded (for filters like Trim).

As could be seen, such architecture not only would allow efficient and scalable utilization of hardware threads but will be able to incorporate even extremely threading-unfriendly filters with sequential frame processing without any correctness or speed drawbacks given enough other work in script.

Also it would be very memory-efficient: only actually needed data is kept and memory is consumed as much as needed for no redundant work in the scheduling window.

Cons: hard to implement, scheduling is active and will consume some resources, how filters work needs to be modified: source frames are provided to filter, not requested from inside.


Legacy Avisynth 2.5 filters work through separate filter "wrapper". Threading information is provided to it in textual user-editable file config.

1) As you said, Avisynth right now is "Render->FilterB->FilterA->Source". Multithreading in Avisynth+ is being implemented by letting multiple such (temporally different) chains be evaluated in parallel. I can't say yet how good it works though, as it needs more work I don't have results yet. Can you describe in more detail what advantage your scheduler would provide compared to a traditional threadpool?
2) That is not possible with current Avisynth. To make this work, all existing plugins would need to be modified (though rather trivially). Anyway, if you do that, you can throw out all exisitng plugins until they are changed, or you can use a wrapper layer which throws away the advantages of your system. Btw, VapourSynth does exactly lthis. It lets plugins specify precisely which frames they need for the current job without the thread having to wait for their availability. This is what allows VapourSynth to implement pipelining, which has its advantages, but also drawbacks IMHO.
3) This is not really a new thing. MT-mode is allowed to be provided by filter itself in both VapourSynth and Aisynth+ (I'll write about that soon in another post), and the modes are already established in all projects.
4) This is similar to what Avisynth+ will allow: to have several frames being processed in parallel, all potentially from different stages of the filterchain. And I think this is possible in VapourSynth too, though I don't know enough of it in this respect to be sure.
5) No cache and strict refcounting: The first is bad for performance, the second is not possible without breaking plugins. I'm in the middle of adapting the cache and memory management of Avisynth+ for multithreading right now actually. Out of curiosity I have made various tests and measurements along the way here, like seeing how the filter chain performs without caching. Absolutely miserably. Caching is so important, that without it performance drops to only a fraction of what it was. Strict refcounting (throwing out frames the moment their refcount drops to zero) is not posisble without breaking all 2.5 plugins, because they have the relevant piece of code baked in. So you can modify the refcounting code to drop frames when the count reaches zero, but that part will not execute in 2.5 plugins. The result is that you will leak every single video frame (possibly multiple times, once for each stage), unless you only use 2.6 plugins, otherwise you'll run out of memory in 100-200 frames using a 720p video. I have found that out the hard way, trying to debug said memory leak for more than a day. FYI, this is exactly why I need to rethink my concept of the new caches in Avs+, which leads to MT being delayed somewhat. A possible "solution" to this refcount problem is to recompile all plugins for the 2.6 interface. But there are a lot of plugins out there, and just because you publish recompiles does not mean the users will not keep using old 2.5 plugins. And we don't even have the source for all plugins (even though we do for the most). All in all, I'm not gonna pursue this, and will instead just implement a different architecture that will work with existing plugins.

[SEt]

1) Scheduler is "advanced" threadpool. Instead of throwing many threads with random caching on hardware and hoping it would work faster, scheduler do exactly the needed work.
2) Like I said, "wrapper" plugin can provide the required additional information, no problem for most filters. In worst case it'll do fail-rerequest route.
5) Cache is absolutely meaningless waste of space if you already can provide all required source frames for the filter call. General refcounting is done by scheduler based on 2, wrapper can do additional refs in response to 2.5 frame refs, but I guess it won't be used much.

[ultim]

Is my understanding correct that you'd let each single filter work on multiple frames in parallel in its isngle thread? And multiple of such superscalar filters would be executing on each HW thread.

[Myrsloik]

I doubt anyone it going to implement it, but as you requested, general outline:
...

Since there may be some interest in comparing approaches and ideas...

VapourSynth works kinda 80% like what you describe.

1. There's scheduling, all frame requests are basically put in a queue and processed in order using a thread pool. If a request can't be processed at the moment (the filter instance is already busy) then it simply looks at the next request in line.

2. VapourSynth filters basically work like this. On the first call they request all (or at least most) frames necessary. Then the filter gets called again when they are ready. The procedure can be repeated if necessary.

3. Exactly

4. You get kinda an implicit, cheap version of this in VapourSynth. It combines requests for the same frame (even before looking in caches) so in many cases it ends up working very well. Even temporal filters normally have a very low number of requests to the actual caches.

5. If filters can do unexpected things you need some kind of cache. Of course 80% of the ones automatically inserted will just be wasting their time doing nothing much at all. And remember that users can also do unexpected things such as seek around in vdub. I do however think avisynth wastes obscene amounts of ram for its caching. Frames have strict reference counting.


Avisynth compatibility works like you say, only difference is that the magic "prefetch list" is compiled in. The method of threading for avisynth filters is a single instance running in its own avisynth environment wrapper. So there are always as many "avisynth environments" as there are filters. In combination with the prefetching each filter instance can at most fully use one cpu core (which usually turns out to be enough since stuff like mvtools is a gazillion medium complexity filters stuck together).

[SEt]

ultim
No, superscalar is scheduler. Its execution pipelines are hardware threads, executed "instructions" are filter instances (according to their threading capability) that work on "registers" – frames.
The key points are breaking the call chain and refcounting instead of cache.

Myrsloik
I'm not interested in VapourSynth.

[TurboPascal7]

So in the end it all comes to the "we need to modify all existing plugins" problem. "Nice".
I guess we'll have to set up a huge repo for all the (useful) plugins ever created and start making them less bad some time soon. I just really want to hope I won't be the only one working on this.

[ultim]

So in the end it all comes to the "we need to modify all existing plugins" problem. "Nice".
I guess we'll have to set up a huge repo for all the (useful) plugins ever created and start making them less bad some time soon. I just really want to hope I won't be the only one working on this.

Well this wouldn't be the first thing we could use such a repository for. Making plugins MT-compatible, recompiling for 64-bit, and checking ASM for calling convention violations would also all require a plugin source repository anyway. Not to mention, it would allow us to get rid of plugins with baked-in code, legacy code in Avisynth's core could be removed, and I could probably also implement a more efficient (and less leaky?) caching.

So there's a real bunch of reasons for setting up a repo for plugins, but there are some basic questions to be answered:

How would plugin authors react? If they are not interested in commiting to the repos in the future, will we have to port every single change they make?
How do we assign plugin maintainership? Obviously we must give authors "VIP" access, and divide the rest of the work between us.
What's the destiny of closed-source plugins?
Who decides which plugins are "useful"? "Useful" depends on who you ask, and what the plugin is used for. And just because some use cases are rare, it doesn't mean that the corresponding plugins aren't useful.
Is it realistic to work on most of the plugins and make all the changes? The changes necessary per plugin are small (unless you have to rewrite asm), but multiplied by the number of plugins...
How do we make sure that a user doesn't use legacy plugins mistakenly after that in Avisynth?
What happens if we fail for whatever reason? Can we depend on plugins being updated in a reasonable amount of time? Should development directions depending on the outcome be halted or not?
Do we have the manpower/time/resolve?


IMHO these are important questions that we should try to answer appropriately before doing anything rash. On the other hand, if we do can handle these question correctly, I'm all for it.

[TurboPascal7]

Wow, stop ruining my dreams so fast...

How would plugin authors react? If they are not interested in commiting to the repos in the future, will we have to port every single change they make?
How do we assign plugin maintainership? Obviously we must give authors "VIP" access, and divide the rest of the work between us.

We do not touch actively maintained plugins. If the author wants to continue using zip packages attached to doom9 posts as his version control system - it's "fine". Our problem is making him "want" to update his plugin for better integration with avs+. The only way we could do it is making sure any updates do not break compatibility with other versions of avisynth. For example asking the author to provide another registration function like AvisynthPluginInit is okay. Asking him to use internal avs+ threadpool is not unless he's really fond of the idea or prepared to maintain a few versions of the plugin.
What's the destiny of closed-source plugins?
Are there many useful of those? I never actually bothered to check what's closed and what's not.
But well, reverse-engineering is pretty much the only thing we could do about these. Not that you don't have to do the same with some open "source" plugins... Of course anything large and complicated is out of questions.

Who decides which plugins are "useful"? "Useful" depends on who you ask, and what the plugin is used for. And just because some use cases are rare, it doesn't mean that the corresponding plugins aren't useful.

What users request. I have maybe 15-20 I use the most, other people use other etc. Yes we'll probably end up with basically all plugins ever created. Can't be helped.

Is it realistic to work on most of the plugins and make all the changes? The changes necessary per plugin are small, but multiplied by the number of plugins...
Simple stuff like recompilation and maybe updating to a newer version of avisynth interface - yes, sure. Rewriting the asm, handling multithreading and making stuff cross-platform - basically impossible unless you find a large dedicated team for that. Hint: you won't.

How do we make sure that a user doesn't use legacy plugins mistakenly after that in Avisynth?

We can't and we don't. Centralizing plugin distribution is pretty much the only thing we can do, but even then no one can stops the user from installing broken plugins from other sources.

What happens if we fail for whatever reason? Can we depend on plugins being updated in a reasonable amount of time? Should development directions depending on the outcome be halted or not?
No, most plugins will never be updated or ported, especially in reasonable amount of time. No, development directions should not depend on this and we should find a way to make things work with what we get right now, at the same time trying to make the most use of plugins that do get updated. Yes it won't be efficient "for now" but it's the only choice we have. I might be able to update some useful plugins which I already worked with and a few more, but I'm a human too.

What we can and should do is making the most popular plugins used in literally 100% of scrips better, so most users get the most benefit from our work. But we cannot save the whole avisynth ecosystem.

[SEt]

So in the end it all comes to the "we need to modify all existing plugins" problem. "Nice".
Of course interfaces must be changed to support new things. It's absolutely normal considering the state of Avisynth interface today.
Binary compatibility with old 2.5 plugins would still be maintained – wrapper is exactly the thing for that. It would cost some efficiency and will require handwriting threading information, but nothing like recompilation required.

Even with other design plans I still suggest the "wrapper" idea: separate from the core plugin that provides compatibility layer. No need to integrate legacy into the new core.

[ultim]

So basically, I can't count on any of the interface improvements made in 2.6. "Great." I can always use wrappers for any kind of 3rd, 4th, 5th interface, but that would still mean keeping around old code, except maybe in its own module. Not to mention now I'd even have to make sure the "glue" logic also works. So thanks, but no. In this case I'll rather continue what I was doing sticking to 2.5-style, and devise *one* new interface for Avisynth+ in the future, when the more important stuff are out of the way, and I can make sure that it will finally be alright. I am not in the mood of maintaining 4+ (redundant!) interfaces, and I don't think I ever will be. But to avoid misunderstandings, plugins should of course still use the 2.6 header. There is currently nothing better than that, as it is most feature complete, and does not suffer from the "baked code syndrome".

... General refcounting is done by scheduler based on 2, wrapper can do additional refs in response to 2.5 frame refs, but I guess it won't be used much.
Refcounting is nice and I would want to use it myself, but I really think I can't because of 2.5 baked code. You can supply the cleanup logic in the users of the frame classes instead of in the frame class, sure, so you could think that might work as long as the core always has the last reference to any frame, instead of the filters. The problem is, how will you know when the refcount reaches zero? The baked-in code won't tell you, so you are left with the choice of either periodic scanning, or with the choice of scanning only when needed, for example on out of memory, which is what current Avisynth does.

You say that we can do refcounting our own way however we want, and provide a wrapper for 2.5 plugins. That means we still have to duplicate the extremely complex old-style Avisynth caching, except in a wrapper, but in addition we must also develop and maintain a second caching system, the primary one?! This is just not worth it, because the old cache basically works. Yes it is way too complex for what it does. Yes it eats memory. But it works and it is not a performance bottleneck. So what reason is there to develop a completely new one if we still have to continue living with all the drawbacks of the current cache, except in a wrapper?

[SEt]

2.6 interfaces are useless – they solved nothing while breaking binary compatibility, usage is very low. I'd say implement only 2.5 compatibility.
New interface obviously needs to be the only one and extensible. Btw, this implies that it must be C style and not the C++ hell like now. For exact design I'd say COM-like: great scalability and linking compatibility.

As for wrapper complexity – it greatly depends on which and how efficient you want it to support quirks of 2.5. Sure, maybe 2.5 plugin that formatted your hard drive would require total emulation to work correctly, but do we care? I suspect that quite simple wrapper + some special cases would be ok for most interesting plugins.

And FTY, caching is the main problem of Avisynth MT. Not threading hacks, not 32-bitness how quite a few people think. And it is performance bottleneck.

[ultim]

2.6 interfaces are useless – they solved nothing while breaking binary compatibility, usage is very low. I'd say implement only 2.5 compatibility.
New interface obviously needs to be the only one and extensible. Btw, this implies that it must be C style and not the C++ hell like now. For exact design I'd say COM-like: great scalability and linking compatibility.

As for wrapper complexity – it greatly depends on which and how efficient you want it to support quirks of 2.5. Sure, maybe 2.5 plugin that formatted your hard drive would require total emulation to work correctly, but do we care? I suspect that quite simple wrapper + some special cases would be ok for most interesting plugins.

And FTY, caching is the main problem of Avisynth MT. Not threading hacks, not 32-bitness how quite a few people think. And it is performance bottleneck.

Well I'm rewriting caching and memory management from scratch anyway. Coz I agree that current cache doesn't lend itselft to MT naturally. In single-threaded mode though it didn't seem like a bottleneck, as it didn't even show up in profiling results.

I also agree that new interface needs to be C-style. My plans are to make it object oriented (as far as C allows), but keep objects totally opaque. An extremely thin C++ wrapper shipped as public include files will provide C++ binding, but it will only provide syntax sugar and some help in memory management thanks to destructors.

[ryrynz]

That have been said, I'm busy with other areas atm. (And you will likely see something useful for Avisynth in several days. ;))

What is it?

[ultim]

You have probably noticed that this release of Avisynth+ is taking longer than previous ones. The good news is that not because we are stalling, but because the features we are working on are a lot larger, and also a lot more complex.

Other good news is that people have started joining in on the project. TurboPascal7 and innocenat have been working on bringing you full-fledged 64-bit functionality, qyot27 has already sent a pull request to give you a nice automated installer, and I myself am introducing multi-threaded processing.

So, look forward to the next release!

[turbojet]

Looking forward to 64 bit and multithreading. A few questions:
1. Will the installer be able to extract the dll's with 7zip?
2. Will the dll package still be available?
3. At some point in the future, would JincResize (from madvr) be considered?

[innocenat]

3. At some point in the future, would JincResize (from madvr) be considered?

I have been considering adding Jinc support (as I also want them), but I have trouble finding reference to this filter. I'd be grateful if someone could link a nice explanation of how this filter works.

[turbojet]

There was a lot of talk about it in the madvr thread last year. I believe it's based on: http://www.imagemagick.org/discourse-server/viewtopic.php?f=22&t=21415 there's some code on page 2 and maybe more in later pages. This might be part of it: http://forum.doom9.org/showthread.php?p=1595043&highlight=sigmoid#post1595043

But only madshi and maybe NicolasRobidoux would really know.

[Keiyakusha]

For Jinc, you also need madshi's anti-ringing, it sux without it.
Also if someone is going to implement it, it will be cool if needed calculations can be done in GPU. Unless it'll be comparable to spline36 by speed. Cause if it'll be slow, one could just go with nnedi/eedi+subpixel shift(if desired) and get better results. And Jinc is also not something you're going to use for downscale.

It could be way simpler if madshi could just turn that part of his renderer into a plugin (even if closed-source).

[TurboPascal7]

Moving stuff to GPU and back is not worth it if resizing is the only thing you're going to do. So no, this is not going to work.
Also, fmtconv might get ported to avisynth some time in the future. We might as well wait for it before spending our time on something that might end up replaced by fmtconv.

[Keiyakusha]

There were examples of gpu resampling in avisynth and they worked quite well (old and not compatible with recent avisynth versions). I imagine today with better gpus and better drivers it'll be fine too. I guess one should try some simple resampling and see how it goes.
Edit: of course it is probabably pointless for something like bicubic, but JincAR is gonna be slowest resampling yet.

[TurboPascal7]

It's not like you can't implement a GPU resizer in avisynth, it's that performance improvement you might get out of it is not worth it in most cases unless your resizer is really complex and slow to do on CPU. I don't think it's a good idea to include slow filters like that into the core.

Things might change if avisynth suddenly becomes GPU-aware, but I'm not aware of any efforts/plans in this area.

[Keiyakusha]

The thing is, I don't care about GPU that much, but rather...
All filters can be roughly divided into 2 groups: fast, average speed - your "go to" filters and slow filters - when you don't care about speed but want to get the most out of it. If it'll turn out that Jinc+AR falls into the second group, there will be no reason to prefer it over nnedi. But at this point probably no one except madshi can estimate possible speed. Maybe even madshi doesn't know, unless he tried to implement it in CPU 1st. On the other hand, unlike nnedi, it was proven that JincAR can be done in GPU and works quite nice. This is where my "make use of GPU" idea comes from. Plus this way madshi probably will be able to give better assistance.

[ultim]

Looking forward to 64 bit and multithreading. A few questions:
1. Will the installer be able to extract the dll's with 7zip?
2. Will the dll package still be available?
3. At some point in the future, would JincResize (from madvr) be considered?

1. Not sure what you mean. Probably no, but cna you clarify?
2. Haven't decided yet. Is there any use to a zip if you can update an existing installation by just simply installing over it?
3. I won't work on it personally, if anyone is interested in porting it to Avisynth though, that'd be cool.

[ultim]

The thing is, I don't care about GPU that much, but rather...
All filters can be roughly divided into 2 groups: fast, average speed - your "go to" filters and slow filters - when you don't care about speed but want to get the most out of it. If it'll turn out that Jinc+AR falls into the second group, there will be no reason to prefer it over nnedi. But at this point probably no one except madshi can estimate possible speed. Maybe even madshi doesn't know, unless he tried to implement it in CPU 1st. On the other hand, unlike nnedi, it was proven that JincAR can be done in GPU and works quite nice. This is where my "make use of GPU" idea comes from. Plus this way madshi probably will be able to give better assistance.

Briefly flying through what's in the first post of here (http://www.imagemagick.org/discourse-server/viewtopic.php?f=22&t=21415), the nice thing is it looks like you could implement the filter on the GPU without any of the CUDA or OpenCL GPGPU BS. Possibly all you need is OpenGL with GLSL and FBO extensions. As a result you could use the filter even on older HW, without any special drivers or dependencies, and even be portable to Linux. My investigations will stop here though, and Turbo is right that a GPU implementation might not pay off due to bouncing textures on and off the GPU.

[Keiyakusha]

Briefly flying through what's in the first post of here (http://www.imagemagick.org/discourse-server/viewtopic.php?f=22&t=21415), the nice thing is it looks like you could implement the filter on the GPU without any of the CUDA or OpenCL GPGPU BS. Possibly all you need is OpenGL with GLSL and FBO extensions. As a result you could use the filter even on older HW, without any special drivers or dependencies, and even be portable to Linux. My investigations will stop here though, and Turbo is right that a GPU implementation might not pay off due to bouncing textures on and off the GPU.

I see. In other words, it is better just leave it alone and do something else. Because GPU implementation might not pay off due to bouncing textures, CPU implementation might not pay off due to nnedi being much better choice.
BTW that discussion you link to is TL;DR but I think it does not includes anti-ringing which is something madshi made on his own. Also I remember madshi was saying something that he tried to implement sigmoidal algorithm but didn't liked it, so his Jinc might be something a bit different.

[ryrynz]

Would it be easy to allow for filters that currently work only on YV12/NV12 to work with PO10? And if that was the case, would anyone be willing to hack ffdshow a bit to allow untouched 10bit frame serving
for the processing of that content? Perhaps even a ffdshow raw replacement for interfacing with Avisynth so that 10bit content can be enhanced without any color space conversion?

[qyot27]

Looking forward to 64 bit and multithreading. A few questions:
1. Will the installer be able to extract the dll's with 7zip?
2. Will the dll package still be available?
1. 7zip can unpack the installer, yes. It's just an update of the NSIS script that classic AviSynth uses for its installer.
2. I would wager any third-party builds would still use a non-installer-based package. So, probably?

The big caveat here is that the installer is assumed to be used by those choosing to rely only on AviSynth+, such as first-time users. It installs to %ProgramFiles%\AviSynth+ and sets the plugins and plugins+ directory targets in the registry to the ones residing in the AviSynth+ directory. This means that - unless some other solutions arise to allow classic and avsplus to coexist - you should only use the installer for either classic or avsplus, but not both. Using the avsplus installer over top of the classic one will more than likely reset the plugin directory locations in the registry and that means you'd have to move/copy the plugins over to AviSynth+'s plugins directory (and if you ever uninstalled AviSynth+, you'd have to be sure to either reinstall classic or fix the registry so it points at AviSynth 2.5\plugins again).

Currently, it cannot install both 32-bit and 64-bit versions, so separate installers for each one would be necessary unless/until a proper solution for that is found. I saw some references on StackOverflow to getting NSIS to use one installer for both, but since I don't have access to any Win64 setups I can't really do any work on that and test it (the closest is 64-bit Wine, but that still means nothing because I can't compile AviSynth+ itself as 64-bit - unless there's some CMake trick to make a Win64 target available to a 32-bit MSVC...if cross-compiling 64-bit stuff with 32-bit MSVC is even possible, that is).




One thing that I have been thinking about in regard to licensing as SEt brought up way earlier in the thread: since several parts of the AviSynth+ source code would be brand-new, I propose that anything brand-new could be put under a more permissive but still GPL-compatible license. The ISC license (http://en.wikipedia.org/wiki/ISC_license) and Expat license (http://en.wikipedia.org/wiki/MIT_License) are both popular and GPL-compatible, for instance. Arguably, the refactoring process on the old stuff might eventually render many of those parts completely rewritten and open to relicensing, but since IANAL I won't say anything definitive on that front.

The end result is that AviSynth+ (in binary form?) would still be distributed under the terms of the GPL (with the plugin linking exception intact), but at least on those brand new parts, it could encourage more source contribution from those that would otherwise not because of the GPL.

[TurboPascal7]

Would it be easy to allow for filters that currently work only on YV12/NV12 to work with PO10? And if that was the case, would anyone be willing to hack ffdshow a bit to allow untouched 10bit frame serving
for the processing of that content? Perhaps even a ffdshow raw replacement for interfacing with Avisynth so that 10bit content can be enhanced without any color space conversion?

No, it won't be easy.
No, no one will be hacking ffdshow since the project is dead. We do have a few ideas about avisynth in directshow, but that will wait.

Oh and I agree on the license part.

[qyot27]

Would it be easy to allow for filters that currently work only on YV12/NV12 to work with PO10? And if that was the case, would anyone be willing to hack ffdshow a bit to allow untouched 10bit frame serving
for the processing of that content? Perhaps even a ffdshow raw replacement for interfacing with Avisynth so that 10bit content can be enhanced without any color space conversion?
I imagine that high bit depth support in general is a long-term goal, and most likely will emerge as 16-bits first, and then fill in with the in-between pixel formats later. Trying to specifically hack 10-bit support into the current codebase is probably not feasible, and anything described as a hack is certainly not desirable.

[ultim]

The big caveat here is that the installer is assumed to be used by those choosing to rely only on AviSynth+, such as first-time users. ...

true, but when i get near a release, i will try to modify the installer to:
- prevent installation if it detects classic avisynth already installed, and give meaningful instructions to the user.
- install both 32- and 64-bit versions (on a 64-bit OS)
- install the required vc++ runtime unattended if needed

qyot's work is a very good start on an avs+ installer, and even if none of the above points get realized right now, his installer can still be used to make releases. but i will try to implement my points soon based on his work, coz i find them important for a seamless user experience.

my experience lies in Wix rather than in NSIS though, so someone more experienced with NISIS could probably do it in half the time than me. any volunteers?

[ultim]

the licensing issue might be worth its own topic. it's a complicated thing, both in practice, both legally, and both in reaching a consensus between all past contributors. i'm all for changing the license, but it only makes sense if we can do it legally properly. otherwise there is little point in having one at all.

[turbojet]

Extractable installer makes standalone dll package not very useful, at least for me. The reasoning behind it is portability or being able to use/update avisynth without admin rights.

Nnedi is limited to multiples of 2 resizing, which doesn't make it a replacement for a general resizer. While it's subjective I think Jinc performs just fine without anti-ringing, imo it's does a better job of detail retention. I can't find it now but in another thread I had asked madshi about Jinc downsizing and he said it could be effective just not implemented. I'm not sure it would be more effective than lanczos when encoding however, during playback lanczos seems to be stay closer to the source.

DGNV's gpu resizing is very fast but I believe it's baked in cuda code and limited to bilinear. It would be interesting to see if a resizing is faster with gpu pixel shaders or cpu.

[innocenat]

I agree that license is complicated thing, and while many parts of avs+ are rewritten and could be put under, say , LGPL, many part also don't. At least on core filter I have been working on, the C++ files are rewritten but header file mostly remain the same. And it's very hard to track which piece of code are written by whom.

[ultim]

I see. In other words, it is better just leave it alone and do something else. Because GPU implementation might not pay off due to bouncing textures, CPU implementation might not pay off due to nnedi being much better choice.

Well it depends on if you are willing to take the risks. Ofc a GPU implementation will be faster if the time lost by copying textures over the buses is a lot smaller than the time won by the GPU processing. If the filter it very slow, than GPU processing will probably still be a good choice. But this means whether it is worth it or not also depends on the difference in speed between your CPU and GPU. It is well posisble that someone with a slow-ish CPU but modern GPU will see a huge benefit, while another guy with a fast CPU but non-gaming (or old) GPU will be hurt by GPU processing.

I agree that license is complicated thing, and while many parts of avs+ are rewritten and could be put under, say , LGPL, many part also don't. At least on core filter I have been working on, the C++ files are rewritten but header file mostly remain the same. And it's very hard to track which piece of code are written by whom.
Exactly. Even if large parts have been rewritten, it is not practical to change the license without permission from previous authors, because without permission, we'd have to track every line which license applies for that particular line number.

On the other hand, the current "GPL+public header exception" is practically the same as the accepted interpretation of the LGPL, so we might be able to pull that change off. It will only work with LGPL though, not with other licenses like MIT. Also note that the 2.6 header is not under the exception right now.

To summarize, trying to get ALL the code under LGPL could be possible, but we'll need other opinions on the matter than just mine, coz I'm not sure if even that is okay. Any other target license surely requires getting in contact with all the previous contributors.

[TurboPascal7]

I'm perfectly fine with LGPL or any other more permissive license, if that matters.

[malmsteen81]

hi, with avisynth+ can i use the filter without setmode's istruction (for multi thread)? right?

[ryrynz]

Avisynth+ is not multithreaded yet, there are no MT modes.

[andybkma]

Greets, can Avisynth+ also be used for realtime script post processing with ffdshow raw filter? Reason being, I swapped out the dll file with the + version and now I get a conflict with mVR which didn't happen before when using the "official" 2.6.0 Alpha 5 avisynth.dll file.

The problem is that I get an error code: "madVR reports: resetting Direct3D device failed 8876017c" when coming out of mVR Fullscreen Exclusive Mode back into window mode after playing a video in FSE for more that 10minutes or thereabouts. Not sure what the cause is, all I know is that it happens when using the avisynth+ dll file and not with the official. Just thought I would let you know. Cheers

AviSynth+ 2013.10.16 + mVR Deband 14 version

[qyot27]

A couple of suggestions for the plugin writing tutorials and/or general explanation about the core: an explanation of what needs to be considered in order to support 64-bit properly, and how to deal with the topic mentioned above with moving to being high bit depth aware (also if relevant to the HBD discussion, anything about adding new pixel formats and/or colorimetries). I'm certainly interested in knowing what the nuts-and-bolts of these are, even if only for educational purposes.

[ultim]

Greets, can Avisynth+ also be used also for realtime script post processing with ffdshow raw filter? Reason being, I swapped out the dll file with the + version and now I get a conflict with mVR which didn't happen before when using the "official" 2.6.0 Alpha 5 avisynth.dll file.

The problem is that I get an error code: "madVR reports: resetting Direct3D device failed 8876017c" when coming out of mVR Fullscreen Exclusive Mode back into window mode after playing a video in FSE for more that 10minutes or thereabouts. Not sure what the cause is, all I know is that it happens when using the avisynth+ dll file and not with the official. Just thought I would let you know. Cheers

AviSynth+ 2013.10.16 + mVR Deband 14 version

Hi!
Yes, Avisynth+ should work fine for that, just like Avisynth did. In fact, I'm often using MPC-HC+ffdshow raw renderer for my own visual tests, so I know for sure you can use it that way for real-time processing. The only difference is I'm not using madVR, but I don't see yet how the changes in Avisynth+ compared to classic Avisynth could cause a Direct3D issue with the renderer.

If you are really sure it is caused by Avisynth+, I'll try to reproduce it when I finish working on threading, then I'll be able to tell more. Until then can you please enter it as a bug report on GitHub so that I won't forget it?

[Myrsloik]

I have a small avisynth.h change request. Feel free to ignore it if you're not going to break the ABI when going 64bit.

Would it be possible to extend VideoFrame:: offset, offsetU and offsetV to be a 64 bit type (such as ptrdiff_t)?

This would make wrapping to VapourSynth internals possible for the 64 bit ABI.

[ultim]

I have a small avisynth.h change request. Feel free to ignore it if you're not going to break the ABI when going 64bit.

Would it be possible to extend VideoFrame:: offset, offsetU and offsetV to be a 64 bit type (such as ptrdiff_t)?

This would make wrapping to VapourSynth internals possible for the 64 bit ABI.

Hi, I have already wanted to make such and similar changes on many occasions, but I resisted, because I really want to keep full ABI compatiblity to existing 64-bit plugins. When the interfaces get updated though, you can be sure that the new interface will use such types. But not in avisynth.h though.

[osgZach]

I have to say I commend your dedication to this effort and I wish you lots of success. I will be watching this (fork?) with interest.

[Jeroi]

So last post was not funny enough, Ultim I am referencing to your desire of modernize the Avisynth into new era so since AVI is kinda oldish video container format (who uses it anymore?) why not Ultim change your Avisynth+ name into more modern name Like Videosynth or Mediasynth or let say Mediaframer or similar more universal name for supporting more modern HD video container formats and the age we live in 2013 where no AVI's anymore?

[Guest]

So last post was not funny enough, since AVI is kinda oldish video container format (who uses it anymore?) why not Ultim change your Avisynth+ name into more modern name Like Videosynth or Mediasynth or let say Mediaframer or similar more universal name for supporting more modern HD video container formats? It's a fork of Avisynth, hence the name. And BTW Avisynth supports modern HD container formats quite fine.

[Jeroi]

It's a fork of Avisynth, hence the name. And BTW Avisynth supports modern HD container formats quite fine.

You obviously missed the point hence there where that reference call in that last post.

[ryrynz]

Could we not continue with this discussion please?

[andybkma]

If you are really sure it is caused by Avisynth+, I'll try to reproduce it when I finish working on threading, then I'll be able to tell more. Until then can you please enter it as a bug report on GitHub so that I won't forget it?

Yep, definitely sure as I just reproduced it again. Don't know what a github is though I am sure if the bug is important to you you won't forget about it ;-)

[StainlessS]

I dont know what a github is either but, see here, https://github.com/pylorak/avisynth

looks like you need to access 'issues' on Right Hand Side of page to give a bug report.

[StainlessS]

Although if you go to above link and click on "New Issue", you get a "404, This Is Not The Page You Were Looking For"
type error message, ie it dont work.

[ajp_anton]

Feature request: allow comments on split lines, like this:
filter(argument 1 #comment
\, argument 2)Another closely related request:

Example:

function1()
#\.function2()
\.function3()

allow commenting out the middle line for easy temporary removal of the middle function. /**/ commenting just isn't very practical...
The previous request would allow for a
\#.function2()
comment, but commenting out the full line including the \ would make life even better =).

[ultim]

Yep, definitely sure as I just reproduced it again. Don't know what a github is though I am sure if the bug is important to you you won't forget about it ;-)

I entered it into the bugtracker: https://github.com/pylorak/avisynth/issues/11

StainlessS: @404, That's becasue you weren't logged in?

ajp_anton: Got it, it will be handled together with your previous request.

[StainlessS]

StainlessS: @404, That's becasue you weren't logged in?


Thank you Ultim.
I actually tried it to see if you needed to be register to give bug report, the 404 made me think that it was not a registration issue.
(EDIT: With that error code, I would not have bothered to register if wanting to report an error, as it looked pointless if the site did not work)

[TurboPascal7]

I actually tried it to see if you needed to be register to give bug report, the 404 made me think that it was not a registration issue.
(EDIT: With that error code, I would not have bothered to register if wanting to report an error, as it looked pointless if the site did not work)
Good news: this is fixed now.

[zero9999]

In preparation for the upcoming release, we are in need of a few translated strings for the AviSynth+ installer. If you want to help, please check out this pad (https://pad.riseup.net/p/gfyPtMbihf6q)

Edit: an in case you want to be credited, please leave your name here, because apparently the pad doesn't save the authors names after they close the window...

[ultim]

Hooray, we've got a couple of announcements to make. All good ones ;)

A new release
Yes we have a new release and a cool one that brings full 64-bit functionality. But here's the semi-detailed changelog.
- A small number of bugs that were regressions compared to classic AviSynth have been fixed. This includes a bitblt copy error, and a script evaluation error that made it necessary to explicitly give the „last“ clip as input to some rare filters. It was discovered on animate(), but it might not have been the only one. The 64-bit version also had non working versions of Amplify(DB), Normalize, and MixAudio, which should be fixed now.
- tp7 and innocenat have finished porting all built-in filters to compiler intrinsics. This is a truly great accomplishment not only becasue it gives us a fully working 64-bit version, but the previous assembly code (which is now gone) was a large obstacle in reaching linux/osx compatibility too. We are still not cross-platform, but their work has brought us a large step closer. Not to mention it also allowed us to get rid of the SoftWire library, which brought down the binary size by 50% (though you might not see this if you compare it to UPX'd versions). I should probably also underline how much work this has been for them, they updated like 19.000 lines of code!
- As yet another consequence of tp7's and innocenat's work, the speed of many internal filters has greatly increased, in some cases 150% or more.
- There is now a shiny new installer. After qyot27's installer update for AviSynth+, line0 brought it another step further and has rewritten the old AviSynth installer from scratch. Compared to the old installer, you not only get a nicer graphical look, but also comprehensive migration options from classic AviSynth too, as well as unified x86/x64 support. Line0 is also working on high-res icons. This is still work in progress, but you can see preliminary results in the installer's icon.

Last but not least, we give special thanks to a random stranger* (see EDIT) who has pioneered in introducing Pig Latin translations to the software world. His work is unfortunately not yet included due to purely technical reasons, but I'm sure that many will appreciate his contribution when we finally do, especially native speakers of Pig Latin.

Homepage and IRC
Also kind of important news is that AviSynth+ now has a homepage, reachable under avs-plus.net (http://avs-plus.net). It is hosted by GitHub and is a bit minimalistic right now, but for sure a better landing page than GitHub's repository dump. There is also a new #avs-plus channel on Rizon for all IRC lovers, and in addition to this forum, you are welcome to influence development of AviSynth+ there too.

And the googly master plan
… is to involve even more people in AviSynth's future. The past weeks have proven to me that I'm far from alone caring for AviSynth, but there are still quite a number of items on AviSynth+'s wish-list, so there is a little thing I'd like to do: enter AviSynth+ into Google Summer of Code (http://www.google-melange.com/gsoc/homepage/google/gsoc2014) as a mentoring organization. GSoC is a wonderful opportunity, as it would not only help in code development, but would spread the word of AviSynth, and give us even more feedback thanks to the larger exposure. I also think it is a fun and rewarding task for students to work on multimedia processing, not to mention they'd even get compensated by Google this time. So we're counting on you, whether it is ideas or volunteering for being a mentor for the project (yes we're looking for mentors, all are eligible that have previously written code or plugins for AviSynth). Also let us know if you're not wishing to get directly involved, but you've got previous experience with GSoC and got a few tips and pointers for us.

Help us spread the word, and live long and prosper.

EDIT: __ar apparently wasn't involved. So we don't know who to thank.

[Jeroi]

I just can't say more than thank you guys. In one day we may have unix native avisynth.

[Guest]

In one day we may have unix native avisynth. There already is one: avxsynth.

http://forum.doom9.org/showthread.php?t=164386

[TurboPascal7]

A few notes on the porting effort, asm and future plans.

1) Myrsloik did not lie - a lot of asm in the core was quite terrible. We actually had to remove HorizontalReduceBy2 YUY2 ISSE implementation because it was slower than the C code. There were some quite good MMX routines though (SSE2 was awful everywhere but resizers). Resizers were good.

2) As mentioned in the first pull request (https://github.com/pylorak/avisynth/pull/12), the general rule was "not slower than original on Nehalem+ CPUs". We did not test on any older CPUs. Expect performance to get a bit worse on Pentiums and I'm not sure about some memory-bound filters on Core 2. Please report if you experience a noticeable performance drop in the core filters on Core 2 level CPUs. We will not be spending a lot of time optimizing for pre-Nehalem CPUs though.

3) All filters now have C versions so you can run them on super ancient CPUs. It will also help non-x86 platform support.

4) All filters now have SSE2 versions. This for example means up to two times faster TemporalSoften. Some also got SSSE3 and SSE4.1 optimizations. You can find which one in the commit messages of the pull requests.

5) There are some behavior changes: TemporalSoften mode 1 is removed, mode parameter is simply ignored. Blur MMX parameter and Tweak SSE parameter are also ignored.

6) MMX optimization routines are dropped if there is a faster ISSE version. This affects only a few filters and some extremely old CPUs.

7) Code from FTurn (http://forum.doom9.org/showthread.php?t=168315) is now integrated into the core (with some additional optimizations and new RGB32 routines), making the plugin obsolete.

8) We did not port MMX code of any audio filters. We won't do this any time soon, feel free to contribute.

9) Resizers are implemented as VerticalResizer().Transpose().VerticalResizer().Transpose() instead of two separate routines for vertical and horizontal resizing. Some rounding differences are possible, although not noticeable. This improves performance in most test cases and simplifies implementation quite a bit.
YUY2 resizer is also implemented as ConvertToYV16().Resize().ConvertToYUY2(). This does not affect performance in any way on the CPUs we were working on. Conversion is lossless and extremely fast.

Speaking of YUY2: support of this color format will be dropped in all external filters we port. We don't have unlimited time or will to work on something that no one uses. You can always process YUY2 with planar filters by converting to YV16 and back.

Now, what next's? Our part of the team will be slowly improving useful external filters to make them work with MT and other platforms in the future. Feel free if you want to help, but please write about it beforehand so we don't end up porting the same filter twice.

Also, I'd like to hear if any of the authors are willing to update their plugins themselves. This includes future MT registration, inline asm removal and later - other platforms support.

EDIT: A note for ICL users. In our tests, ICL14 tends to generate slower code for some filters, e.g. resizers. You might want to check it before publishing any dlls (which I still recommend you NOT to do). I don't know if the same applies to older versions and I don't know what's the older version you can compile avs+ with. For VS it's vc100 right now (2010).

[real.finder]

hi :)

I got

http://i.imgur.com/qJ2gvlZ.png

with "Dither_convert_yuv_to_rgb"

and thank you for this project :)

[zero9999]

hi :)

I got (some masktools error) with "Dither_convert_yuv_to_rgb"

and thank you for this project :)
you're using a masktools version built with avisynth 2.5 header. use this (http://tmod.nmm-hd.org/Misc/mt_masktools-26-for-2.6alpha4.7z)version instead.

[turbojet]

Some first observations:

1. Installer doesn't play nice with avisynth, assigns new plugin directory and doesn't restore old after uninstall
2. Installer installs VC++ redist even though they were already up to date
3. Installer migration option deletes avisynth plugins, luckily they were backed up.
4. Installer isn't extractable, if extractable it's a simple copy/paste of avisynth.dll Wasn't this supposed to be extractable like avisynth installers?
5. Getting all sorts of dll missing with avisynth+, avsrecursion.dll and opencl.dll mostly which are in avisynth+ and avisynth plugin directories.
6. Some speed tests: With a 1080i mpeg2 source with dss2x86/dssx64(lav avcodec decoding).tfm().tdecimate().lanczosresize()

2.6a5icl
720x404 = 37.96 fps (50-60% cpu)
1280x720 = 25.74 (100%)

avs+:
720x404 = 36.98(50-60%)
1280x720 = 24.74(100%)

avs64 (JoshyD)
720x404 = 25.02(30-40%)
1280x720 = 21.80(70-80%)

avs+64
720x404 = 22.84(30-40%)
1280x720 = 19.17(70-80%)

I've been battling with fixing avisynth plugins directory for almost an hour now and still haven't solved it. Guess a rollback is necessary. EDIT: IOBit uninstall of avs+ then clean install of avisynth fixed the plugins directory. Seems the uninstaller doesn't clean up very well.

[zero9999]

Some first observations:

1. Installer doesn't play nice with avisynth, assigns new plugin directory and doesn't restore old after uninstall
3. Installer migration option deletes avisynth plugins, luckily they were backed up.


Which upgrade option did you pick? The second one (migration) is designed to do exactly what you described in [1]. It also does [3], but only after copying all your plugins to the new plugin directory for 2.5 plugins.

Some first observations:
2. Installer installs VC++ redist even though they were already up to date

The VC++ Redist installers automatically determine whether or not they need to install/update the runtimes and just do nothing if everything is up-to-date. Are you sure you were already running Update 4?

4. Installer isn't extractable, if extractable it's a simple copy/paste of avisynth.dll Wasn't this supposed to be extractable like avisynth installers?

You can extract it with InnoExtract.


5. Getting all sorts of dll missing with avisynth+, avsrecursion.dll and opencl.dll mostly which are in avisynth+ and avisynth plugin directories.

not sure about this, please be more precise. Avs+ doesn't touch those libs (which should reside in your SysWoW64 folder on 64-bit windows)


6. Some speed tests: With a 1080i mpeg2 source with dss2x86/dssx64(lav avcodec decoding).tfm().tdecimate().lanczosresize()

is this the full script? What tool did you use to get those numbers and where did you get it from?

[turbojet]

Which upgrade option did you pick? The second one (migration) is designed to do exactly what you described in [1]. It also does [3], but only after copying all your plugins to the new plugin directory for 2.5 plugins.
At first the top option, then uninstalled and installed avisynth and plugins weren't loading. Then tried bottom option, avisynth\plugins and plugins64 were moved to empty avs+\plugins plugins64 dir thus deleted. IOBit uninstalled, installed avisynth again and plugin autoloading was working again.



The VC++ Redist installers automatically determine whether or not they need to install/update the runtimes and just do nothing if everything is up-to-date. Are you sure you were already running Update 4?
I don't think it was actually, they dated back to September I believe. Even so it's says installing on reinstall for about 5 seconds, although it might just be a bad message.


You can extract it with InnoExtract.
All I can get out of it is avisynth.dll and devil.dll not sure if it's x86 or x64. Avisynth installers are fully extractable with 7zip. If installer isn't extractable could a zip with the dll's also available?


not sure about this, please be more precise. Avs+ doesn't touch those libs (which should reside in your SysWoW64 folder on 64-bit windows)
These libs were in avisynth+\plugins directory and it was throwing errors with them, works fine with avisynth.


is this the full script?
Yes. The x64 speed decrease is I believe a decoder issue. That's fed through x264 same options.

[real.finder]

you're using a masktools version built with avisynth 2.5 header. use this (http://tmod.nmm-hd.org/Misc/mt_masktools-26-for-2.6alpha4.7z)version instead.

That's weird, I use this version but load it manually, 2.5 version of masktool in autoload folder

it supposed that manual load eliminates the auto one like the normal avs did, not vice versa

[zero9999]

At first the top option, then uninstalled and installed avisynth and plugins weren't loading. Then tried bottom option, avisynth\plugins and plugins64 were moved to empty avs+\plugins plugins64 dir thus deleted. IOBit uninstalled, installed avisynth again and plugin autoloading was working again.

So after you tried the first option (backup) and then uninstalled Avs+, did the uninstaller not restore your previous AviSynth installation, or why did you reinstall AviSynth again after that?
Did you try to install AviSynth+ into your previous AviSynth folder?

As far as IOBit software goes, you'd have to check with them because i don't know anything about what it does.

Since you seem to have restored your previous state, it's kinda hard to investigate the issues but could you please check the following registry keys and post the contents:

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\AviSynth
HKEY_LOCAL_MACHINE\SOFTWARE\AviSynth
HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall\{AC78780F-BACA-4805-8D4F-AE1B52B7E7D3}_is1


I don't think it was actually, they dated back to September I believe. Even so it's says installing on reinstall for about 5 seconds, although it might just be a bad message.

This is normal. The vcredist installers are always run to let them decide for themselves whether they need to install anything or not


All I can get out of it is avisynth.dll and devil.dll not sure if it's x86 or x64. Avisynth installers are fully extractable with 7zip. If installer isn't extractable could a zip with the dll's also available?

<line0> zip build with just the DLLs would be nice too
<&ultim> will do


These libs were in avisynth+\plugins directory and it was throwing errors with them, works fine with avisynth.

Those libraries are not AviSynth plugins so they shouldn't go into the plugin directories in the first place.
Actually this is interesting. You said the Avs+ installer was not copying the contents of the old plugin folders to the new locations, but apparently it did so at least for 32 bit stuff.
am i right to assume that only the migration of the 64-bit plugins failed?


Yes. The x64 speed decrease is I believe a decoder issue. That's fed through x264 same options.
don't use x264 to benchmark scripts. Use AvsMeter insteadl

That's weird, I use this version but load it manually, 2.5 version of masktool in autoload folder

it supposed that manual load eliminates the auto one like the normal avs did, not vice versa
yes, and that's exactly the way it works here.

[TurboPascal7]

real.finder
I just tried your way and it works as expected.

[real.finder]

real.finder
I just tried your way and it works as expected.

If I back to normal avs I have no problem with the same script, I'll do more tests to see what causes this

[TurboPascal7]

If I back to normal avs I have no problem with the same script, I'll do more test to see what causes this
Please paste the exact script you have issues with.

[real.finder]

Please paste the exact script you have issues with.

DGDecode_MPEG2Source("C:\Documents and Settings\a\Desktop\New Folder\One Piece 517__cut.demuxed.d2v")
LoadPlugin("C:\Documents and Settings\a\Desktop\mt_masktools-26.dll")
Dither_convert_yuv_to_rgb

in winxp sp3 on VirtualBox v4.3.2.90405

[TurboPascal7]

Ok, the issue is related to the placement of the LoadPlugin line. It will work correctly if you place it before the MPEG2Source call.
Yet this is a bug, will investigate. Thanks for the report.

[real.finder]

Ok, the issue is related to the placement of the LoadPlugin line. It will work correctly if you place it before the MPEG2Source call.
Yet this is a bug, will investigate. Thanks for the report.

LoadPlugin("C:\Documents and Settings\a\Desktop\mt_masktools-26.dll")
DGDecode_MPEG2Source("C:\Documents and Settings\a\Desktop\New Folder\One Piece 517__cut.demuxed.d2v")
Dither_convert_yuv_to_rgb

yes, that work :)

[Lenchik]

Those libraries are not AviSynth plugins so they shouldn't go into the plugin directories in the first place.
AFAIK, starting with some 2.6 build Avisynth is searching for additional libraries in plugins folder before system folder.

[the_weirdo]

AFAIK, starting with some 2.6 build Avisynth is searching for additional libraries in plugins folder before system folder.

IIRC, this is only applied to SEt's Avisynth 2.6 MT builds. I don't know if those changes have been merged to upstream yet, though.

[qyot27]

One thing I saw regarding the ISS installer was that it failed to install on an Athlon64 machine I have access to that runs 32-bit XP, complaining that it can't install x64 versions and then quits (why is it trying to install 64-bit on a 32-bit OS?).

Another, more minor thing is that under 64-bit Wine, it installed the 64-bit plugin directories under Program Files (x86) and didn't put anything in Program Files. This might be a Wine vs. real Windows issue, though (especially since it ended up hanging on the vcredist step).

[turbojet]

So after you tried the first option (backup) and then uninstalled Avs+, did the uninstaller not restore your previous AviSynth installation, or why did you reinstall AviSynth again after that?
Did you try to install AviSynth+ into your previous AviSynth folder?

Correct, tried to install avisynth to get the plugin autoloading working again but failed. Never tried installing avs+ into avs install directory.

As far as IOBit software goes, you'd have to check with them because i don't know anything about what it does.

It's like Revo uninstaller, but completely free, it cleans up after uninstaller that don't clean up well.

Since you seem to have restored your previous state, it's kinda hard to investigate the issues but could you please check the following registry keys and post the contents:

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\AviSynth
HKEY_LOCAL_MACHINE\SOFTWARE\AviSynth
HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows\CurrentVersion\Uninstall\{AC78780F-BACA-4805-8D4F-AE1B52B7E7D3}_is1

After installing avs+ with second option, uninstalling, installing avisynth there was some reg entries and the plugin directory was set to the correct path in either hklm\software\avisynth or hklm\software\wow6432node\avisynth but not both like they are set currently in the working state, but it wasn't autoloading. I'm not sure about the uninstall key, it's currently not there in it's current working state. I'll try to break autoloading again in a vm tomorrow if I find time and it's necessary.

This is normal. The vcredist installers are always run to let them decide for themselves whether they need to install anything or not

Would it make sense, if possible, to hide the 'installing vcredist' message in the installer?

<line0> zip build with just the DLLs would be nice too
<&ultim> will do

Thanks


Those libraries are not AviSynth plugins so they shouldn't go into the plugin directories in the first place.
Actually this is interesting. You said the Avs+ installer was not copying the contents of the old plugin folders to the new locations, but apparently it did so at least for 32 bit stuff.
am i right to assume that only the migration of the 64-bit plugins failed?

I'm not sure where avsrecursion came from but I don't use the filters that requires it, at least not often. nnedi3ocl.cl is required for nnedi3ocl() to function. Replacing avisynth.dll with avisynth+'s does not throw the errors. Autoloading other dll's from avisynth plugins makes a lot of sense and makes life easier. A lot of programs load dll's from their directory.

Originally had F:\tools\avisynth\plugins and F:\tools\avisynth\plugins64, when installing avisynth+ with the top install option to F:\tools\avisynth+ it created empty directories of F:\tools\avisynth+\plugins and F:\tools\avisynth+\plugins32 and deleted F:\tools\avisynth\plugins and F:\tools\avisynth\plugins64, the rest of F:\tools\avisynth\ was untouched. The only thing in avisynth+'s default plugins directory were ones in the installer.

don't use x264 to benchmark scripts. Use AvsMeter insteadl

While good for finding bottlenecks, it's not nearly as accurate of a real world test as an encode but here they are (x86 only):

ICL: 720x404 40.46fps (25% cpu) 1280x720 39.97(27%)
AVS+: 720x404 38.75(28%) 1280x720 38.26(29%)

My whole script for x86: dss2().tfm().sorathread().tdecimate().sorathread().lanczosresize()
x64: dss().tfm().tdecimate().lanczosresize()

[ultim]

Looks like there's a confirmed bug in the installer, and as per turbojet's report the DLL search path handling in the plugin loader looks to be inconsistent with classic avs. This wasn't changed though since the very first release of avs+, so that's what you get for not testing early enough :) Furthermore, tp7 confirmed some filter bug in the intrinsics port. Expect a bugfix release tonight (CET).


Would it make sense, if possible, to hide the 'installing vcredist' message in the installer?
It wouldn't. That's what it's doing, so why lie about it? As already said by others, vcredist won't do anything if it is already installed. The 2012 runtime alone has multiple versions though, and the one used by avs+ is the latest that is pretty new. So you probably don't have it already.

[zero9999]

One thing I saw regarding the ISS installer was that it failed to install on an Athlon64 machine I have access to that runs 32-bit XP, complaining that it can't install x64 versions and then quits (why is it trying to install 64-bit on a 32-bit OS?).

yup, already fixed that one. it was actually trying to back up x64 AviSynth files. This (https://files.line0.in/builds/AviSynth%2B-2013-11-24.exe) installer should work


Another, more minor thing is that under 64-bit Wine, it installed the 64-bit plugin directories under Program Files (x86) and didn't put anything in Program Files. This might be a Wine vs. real Windows issue, though (especially since it ended up hanging on the vcredist step).
The installer doesn't create two program directories, even if both x86 and x64 AviSynth are to be installed. Everything goes into the Folder you pick on the 'Select application directory' screen (which defaults to a location in Program Files (x86).

Do you think it would be better to install x64 AviSynth+ to Program Files and x86 AviSynth+ to Program Files (x86)?
IMO it only serves to confuse the users (Where are my plugin directories?) and we'd have to ask for a second Install folder. On top of that it's complicated to pull off because in InnoSetup the components selection screen comes after the select directory screen, so we'd have to do something custom instead.

[real.finder]

Some first observations:
2. Installer installs VC++ redist even though they were already up to date


same thing happened to me, and confirmed it via System Restore

And I saw that c++ 2012 installed many times



<line0> zip build with just the DLLs would be nice too
<&ultim> will do



that very nice :) especially with the new feature in mp_pipeline

[zero9999]

Correct, tried to install avisynth to get the plugin autoloading working again but failed. Never tried installing avs+ into avs install directory.

After installing avs+ with second option, uninstalling, installing avisynth there was some reg entries and the plugin directory was set to the correct path in either hklm\software\avisynth or hklm\software\wow6432node\avisynth but not both like they are set currently in the working state, but it wasn't autoloading. I'm not sure about the uninstall key, it's currently not there in it's current working state. I'll try to break autoloading again in a vm tomorrow if I find time and it's necessary.


just a heads up: i've been able to reproduce and fix the plugin folders issue. please do not try again until ultim posts a new build (which will also include a fix for the autoload issue).

[qyot27]

The installer doesn't create two program directories, even if both x86 and x64 AviSynth are to be installed. Everything goes into the Folder you pick on the 'Select application directory' screen (which defaults to a location in Program Files (x86).

Do you think it would be better to install x64 AviSynth+ to Program Files and x86 AviSynth+ to Program Files (x86)?
IMO it only serves to confuse the users (Where are my plugin directories?) and we'd have to ask for a second Install folder. On top of that it's complicated to pull off because in InnoSetup the components selection screen comes after the select directory screen, so we'd have to do something custom instead.
I think it's potentially confusing to have to put 64-bit plugins under Program Files (x86), even if the folder does say plugins64. If the user knows the 64-bit version of AviSynth+ is getting installed, they'd expect it to be in Program Files and the 32-bit to be in Program Files (x86). At least if they're knowledgeable enough to know that on Win64, 'Program Files' is for 64-bit and 'Program Files (x86)' is for 32-bit.

If InnoSetup can take advantage of it, using an NTFS junction to link the installation directories together might be a solution. That way, there still is only one installation folder, but it's visible in two places that agree with the convention of installing x64 to one place and x86 to another.

[jpsdr]

First, thanks for this project and all of your work.
Just out of curiosity : On some resample filters (i think the spline36 at least), the avisynth64 version took advantage of extra registers avaibles in x64 mode to greatly increase the speed (around 30% if i remember properly). Have you been able to keep this kind of improvement ?

Edit : Just seeing the previous post, and it may have no effect, but remember that Windows XP can be installed on FAT32, so installation partition may not be in NTFS... So, make sure the installer don't take for granted that partition is NTFS (unless you drop XP support).

[TurboPascal7]

Just out of curiosity : On some resample filters (i think the spline36 at least), the avisynth64 version took advantage of extra registers avaibles in x64 mode to greatly increase the speed (around 30% if i remember properly). Have you been able to keep this kind of improvement ?

The code generated for SSSE3, which is the fastest vertical resizer implementation, does not use any additional registers and I don't see where it could benefit from those. SSE2 version does but it's probably still slower than SSSE3.

I have no idea how the final performance compares to avs64, I never checked.

[qyot27]

yup, already fixed that one. it was actually trying to back up x64 AviSynth files. This (https://files.line0.in/builds/AviSynth%2B-2013-11-24.exe) installer should work
It does work. Although, is it from installing AviSynth.dll and DevIL.dll that it requires a restart every time, or from the vcredist?

Edit : Just seeing the previous post, and it may have no effect, but remember that Windows XP can be installed on FAT32, so installation partition may not be in NTFS... So, make sure the installer don't take for granted that partition is NTFS (unless you drop XP support).
Junctioning would only happen on x64 (since x86 doesn't need to bother with it), and this then factors down to:

Probability: The percentage of users that would actually be using Windows XP Professional x64 is ridiculously small amongst the XP userbase because it wasn't widely available to consumers except as an OEM/direct purchase from Microsoft. I don't know if the x64 version will let you install to FAT32, but even if it can, the percentage of users that would do this is even smaller. So a tiny fraction of a tiny fraction of XP users would be affected by the lack of support for junctioning.

At first, I thought that the service pack situation would exclude it since XP Pro x64 never got a Service Pack 3, but it turns out it identifies itself as 5.2, same as Server 2003 (which it was based on).

Regardless, this won't stop anything if junctioning isn't supported on the filesystem; you just have the current situation of everything being in one Program Files location or the other. That is, if you even tell it to install to Program Files at all.

[ultim]

Hello folks, here is the bugfix we promised to you. Issues with the installer are hopefully fixed, and the plugin loader got two patches too for things that have been reported. This (http://forum.doom9.org/showthread.php?p=1655169#post1655169) had to stay off the fix-list though, I didn't even get to look at it due time (but I have a pretty good guess what is going on). I will reach in a fix for that another day, shortly. Until then, enjoy line0's updated installer look and icons.

EDIT: oh I almost forgot, there's a zip-release too.

[ryrynz]

Will we see mulithreading this year?

[TurboPascal7]

Will we see mulithreading this year?
Likely.

[turbojet]

Does the new installer need to be tested for plugin issues I reported?

Also is there some reason why avisynth+ is 5-10% slower with higher cpu usage then avisynth on an fx8320?

[TurboPascal7]

Also is there some reason why avisynth+ is 5-10% slower with higher cpu usage then avisynth on an fx8320?

No one ever tested it on an AMD CPU. What filters? Exact testing scripts if possible.

Also what version of avs you're comparing with (ICL build, 2.5.8., 2.6a5?) and how are you measuring performance?

[turbojet]

http://forum.doom9.org/showthread.php?p=1655141#post1655141
http://forum.doom9.org/showthread.php?p=1655193#post1655193
are the benchmarks, what sort of speed increases are there on intel cpus?

[TurboPascal7]

what sort of speed increases are there on intel cpus?
I don't get any performance difference between the ICL build and avs+ with your script on Nehalem.

Could you please benchmark the resizer call alone on blankclip?

[turbojet]

blankclip(1000,1920,1080).lanczosresize()
ICL: 720x404: 47.99(12%) 1280x720: 39.95 (12%)
AVS+: 720x404 45.67(12%) 1280x720: 32.56 (12%)

[Groucho2004]

blankclip(1000,1920,1080).lanczosresize()
ICL: 720x404: 47.99(12%) 1280x720: 39.95 (12%)
AVS+: 720x404 45.67(12%) 1280x720: 32.56 (12%)

With the same script, on a i5 2500K@4GHz I get this:
blankclip(1000,1920,1080).lanczosresize()
ICL: 720x404: 51.51(25%) 1280x720: 46.80 (25%)
AVS+: 720x404: 74.31(25%) 1280x720: 60.77 (25%)

How about for speed testing we use something more elaborate like this:
colorbars(width = 1920, height = 1080, pixel_type = "yv12").killaudio().assumefps(24000, 1001)

trim(0,499)
fadeio(248)
trim(0,499)

spline36resize(width() - 64, height() - 64).turnleft()
spline64resize(width() + 64, height() + 64).turnright()
bicubicresize(width() - 64, height() - 64).fliphorizontal()
sincresize(width() + 64, height() + 64).flipvertical()

a = tweak(hue=33)
u_chroma = blankclip(utoy(a), color=$808080)
ytouv(u_chroma, a.vtoy)
mergeluma(a)
tweak(hue=-33)

temporalsoften(4,4,8,15,2)
limiter(16, 235, 16, 240)
levels(0, 1, 255, 16, 235)
scriptclip("subtitle(string(ydifferencefromprevious))")
a=selectevery(3, 0).addborders(0,0, 16,0).crop(0,0, -16,0)
b=selectevery(3, 1).addborders(0,0, 32,0).crop(0,0, -32,0)
c=selectevery(3, 2).addborders(0,0, 64,0).crop(0,0, -64,0)
interleave(a, b, c)

I think it has most internal functions, suggestions for improvement are of course welcome. Possibly the GScript component should be added?

[andybkma]

Hello, downloaded newest 2013. nov. 25 binaries, replaced the orig avisynth.dll file (Avisynth 2.6.0 Alpha 5) with the x86 version (and tried the x64 version as well), now when I try to play a video with Zoom Player (with LSFMod as post processing) I get error "Access Violation at 00000000, Read of address 00000000". Putting back the orig avisynth.dll file all is fine again. Note: Previous Avisynth+ version from Sept (?) didn't have this problem...

[TurboPascal7]

Hello, downloaded newest 2013. nov. 25 binaries, replaced the orig avisynth.dll file (Avisynth 2.6.0 Alpha 5) with the x86 version (and tried the x64 version as well), now when I try to play a video with Zoom Player (with LSFMod as post processing) I get error "Access Violation at 00000000, Read of address 00000000". Putting back the orig avisynth.dll file all is fine again. Note: Previous Avisynth+ version from Sept (?) didn't have this problem...

Does it happen with any video or maybe only specific resolution?

[Gavino]

Possibly the GScript component should be added?
GScript (or its equivalent now incorporated into Avisynth+) has no effect on run-time performance, as it is purely a compile-time function. I suppose it could be used to construct more elaborate filter graphs, if that's what you mean.

[Groucho2004]

GScript (or its equivalent now incorporated into Avisynth+) has no effect on run-time performance, as it is purely a compile-time function. I suppose it could be used to construct more elaborate filter graphs, if that's what you mean.
I see. The idea is simply to put as many internal functions as possible into the script for error/speed testing.

[turbojet]

I think the slowdown on amd's is limited to the resizers. Running the second script Groucho2004 posted:

ICL: 4.13 fps
AVS+: 6.21 (33% faster)

No Resizing
ICL: 5.95
AVS+: 9.05 (35% faster)

Just Colorbars+Resizing
ICL: 13.52
AVS+: 18.46 (27% faster)

...and then the blankclip benchmarks above

Unfortunately resizing is about the only internal function I ever use. Would the devs care to look into it? I'm willing to continue testing.

MP_pipeline makes testing the 2 versions much easier.

[ultim]

Will we see mulithreading this year?

Hopefully the last major thing left for MT is making he caches adaptive in their size (and some misc. things). I'm saying "hopefully" because there have already been unexpected challenges in the past, but let's hope most of them are over. The basic elements of MT are all in place now, but the new caches are not yet up to par with the old one. Their problem is that they have a constant and static size currently, so small caches make the encoding slow, large caches cause out of memory. Anyway I'm working on that now in the little free time i have.

[ultim]

Hello, downloaded newest 2013. nov. 25 binaries, replaced the orig avisynth.dll file (Avisynth 2.6.0 Alpha 5) with the x86 version (and tried the x64 version as well), now when I try to play a video with Zoom Player (with LSFMod as post processing) I get error "Access Violation at 00000000, Read of address 00000000". Putting back the orig avisynth.dll file all is fine again. Note: Previous Avisynth+ version from Sept (?) didn't have this problem...

Please send me your script, and it might be also useful if you uploaded a piece of your video somewhere. If the video is long, just a second of it is more than enough.

[ryrynz]

Anyway I'm working on that now

Awesome, great to see so many commits to the project from you guys. How will the MT code in Avisynth+ stack up vs SEt's Avisynth MT?

[jpsdr]

The percentage of users that would actually be using Windows XP Professional x64 is ridiculously small amongst the XP userbase because it wasn't widely available to consumers except as an OEM/direct purchase from Microsoft. I don't know if the x64 version will let you install to FAT32...
Yes it will...:D

[zero9999]

Yes it will...:D
so does Windows 8.1

[jpsdr]

Euh...? :eek: They rolled back ? Because with Windows 7 you can't install it on FAT32 partition, neither x86 or x64. I don't know for sure, but i think it's the same for Vista, you can't install it on FAT32 partition...

[zero9999]

Euh...? :eek: They rolled back ? Because with Windows 7 you can't install it on FAT32 partition, neither x86 or x64. I don't know for sure, but i think it's the same for Vista, you can't install it on FAT32 partition...
just because you can't install Windows on a FAT32 partition, the same is not true for application software.

[TurboPascal7]

Unfortunately resizing is about the only internal function I ever use. Would the devs care to look into it? I'm willing to continue testing.

As I understand it right now (I didn't work on resizers), this performance drop comes from the removed loop unrolling in the innermost resizer loop and branch prediction sucking on your CPU. Previous implementation used softwire to generate this loop but it isn't an option anymore.

We will look into other ways of optimizing it later but our motivation is not exactly high here. As usual - pull requests welcome.

[turbojet]

As I understand it right now (I didn't work on resizers), this performance drop comes from the removed loop unrolling in the innermost resizer loop and branch prediction sucking on your CPU. Previous implementation used softwire to generate this loop but it isn't an option anymore.

We will look into other ways of optimizing it later but our motivation is not exactly high here. As usual - pull requests welcome.

Directshowsource is also much slower, is it for the same reason?

In the meantime maybe MT will give me reason to switch. Will it mt filters like (t)decimate that require frames to come in order?

[TurboPascal7]

Directshowsource is also much slower, is it for the same reason?
No, we didn't touch it. How much slower and what's the test script/source?

Will it mt filters like (t)decimate that require frames to come in order?
Filters that cannot be multithreaded efficiently won't magically become threaded. TDecimate is one of the most complex cases and it most likely won't benefit much if at all.

[andybkma]

Please send me your script, and it might be also useful if you uploaded a piece of your video somewhere. If the video is long, just a second of it is more than enough.

LSFMod script which is here : http://forum.doom9.org/showthread.php?t=142706

Video is anything (using in conjunction with ffdshow raw as post processing filter, no other settings checked in ffdshow raw, just avisynth)

Cheers ;-)

[turbojet]

No, we didn't touch it. How much slower and what's the test script/source?

Nevermind, dss itself isn't slower.


Filters that cannot be multithreaded efficiently won't magically become threaded. TDecimate is one of the most complex cases and it most likely won't benefit much if at all.

TDecimate's developer has said a few lines of openmp (what nnedi3 uses) might work but maybe it was tried without luck.

[TurboPascal7]

TDecimate's developer has said a few lines of openmp (what nnedi3 uses) might work but maybe it was tried without luck.
Nnedi3 does not use openmp, eedi3 does. In any case it does not have anything to do with the core since openmp usage is an implementation detail of a particular plugin. The core will only provide frame-level threading plus maybe some thread pool api just like avstp does.

[ultim]

LSFMod script which is here : http://forum.doom9.org/showthread.php?t=142706

Video is anything (using in conjunction with ffdshow raw as post processing filter, no other settings checked in ffdshow raw, just avisynth)

Cheers ;-)

I already checked LSFMod right after your initial report and it worked without problems. I asked for a complete script and video in the hope of maybe reproducing the problem that way. If uploading the video is not an option, at least tell me your exact video resolution and color space (and full script ofc).

[Groucho2004]

Here (http://www.mediafire.com/download/9jqpn1748qvm9nw/AVSMeter173_x64.rar) is a 64 Bit version of AVSMeter to be used with Avisynth+ 64 Bit. Unfortunately, I only have 4G of RAM so I could not test it beyond that. For you guys with 8, 16 or even 32 G - have a go and let me know if the memory usage that it reports is correct.
A script like this:
setmemorymax(8000)
n = 10000
colorbars(width = n, height = n, pixel_type = "yv12").killaudio().assumefps(24000, 1001)

spline36resize(width() - 64, height() - 64).turnleft()
spline64resize(width() + 64, height() + 64).turnright()

temporalsoften(4,4,8,15,2)
a=selectevery(3, 0).addborders(0,0, 16,0).crop(0,0, -16,0)
b=selectevery(3, 1).addborders(0,0, 32,0).crop(0,0, -32,0)
c=selectevery(3, 2).addborders(0,0, 64,0).crop(0,0, -64,0)
interleave(a, b, c)

should make Avisynth use plenty of memory if you increase "n" to 15000 or more.

[real.finder]

Here (http://www.mediafire.com/download/9jqpn1748qvm9nw/AVSMeter173_x64.rar) is a 64 Bit version of AVSMeter to be used with Avisynth+ 64 Bit. Unfortunately, I only have 4G of RAM so I could not test it beyond that. For you guys with 8, 16 or even 32 G - have a go and let me know if the memory usage that it reports is correct.
A script like this:
setmemorymax(8000)
n = 10000
colorbars(width = n, height = n, pixel_type = "yv12").killaudio().assumefps(24000, 1001)

spline36resize(width() - 64, height() - 64).turnleft()
spline64resize(width() + 64, height() + 64).turnright()

temporalsoften(4,4,8,15,2)
a=selectevery(3, 0).addborders(0,0, 16,0).crop(0,0, -16,0)
b=selectevery(3, 1).addborders(0,0, 32,0).crop(0,0, -32,0)
c=selectevery(3, 2).addborders(0,0, 64,0).crop(0,0, -64,0)
interleave(a, b, c)

should make Avisynth use plenty of memory if you increase "n" to 15000 or more.

nice :)

I'll try it as soon as I receive the new server with 16G ram :)

and I will start with the real encoding

thank you :)

[Wilbert]

- tp7 and innocenat have finished porting all built-in filters to compiler intrinsics.
Just curious. I don't know much about compiler intrinsics. Why compiler intrinsics instead of yasm, nasm or softwire? Can you use those compiler intrinsics on other platforms (once the porting is done)?

[TurboPascal7]

Just curious. I don't know much about compiler intrinsics. Why compiler intrinsics instead of yasm, nasm or softwire? Can you use those compiler intrinsics on other platforms (once the porting is done)?

Just read part 5 of the second guide on this page (http://www.agner.org/optimize/). It has a nice list of pros and cons of using intrinsics. In short: more portable, easier to write and support, but you don't have the full control over your asm (like instruction ordering or register allocation), which means lower possible performance in the best case. In general though, you'll be getting better performance unless you really know what you're doing and prepared to spend hours and hours for optimizing your external asm code.

[TheFluff]

Intrinsics was a terrible idea with VC6 because VC6 ended up generating terrible code from the intrinsics (Avery Lee's blog has a lot of gory details if you are curious). Fortunately nobody is using VC6 anymore (RIGHT?) and current compilers do the Right Thing so intrinsics code is mostly insignificantly slower than writing your own assembler.

[Groucho2004]

Fortunately nobody is using VC6 anymore (RIGHT?)
I could name a few who still use it including myself. I still have a bunch of legacy programs that I wrote ages ago for my employer which were VC5/6 with MFC. I simply don't have the time to port the code to a newer compiler. Also, they just work even though they are built with that "terrible" compiler.

[TurboPascal7]

I could name a few who still use it including myself. I still have a bunch of legacy programs that I wrote ages ago for my employer which were VC5/6 with MFC. I simply don't have the time to port the code to a newer compiler. Also, they just work even though they are built with that "terrible" compiler.

Of course there are programs like that. It does not mean that using VC6 for anything new is reasonable these days.

[wOxxOm]

... let me know if the memory usage that it reports is correct.Tried setmemorymax(12000) n = 20000, got a 12-13gb mem usage, correctly reported, checked against processhacker/procexp.

[Groucho2004]

Tried setmemorymax(12000) n = 20000, got a 12-13gb mem usage, correctly reported, checked against processhacker/procexp.
OK, thanks for testing.

[Groucho2004]

@AVS+ devs
Any chance you could change the version string (from internal.h) to something more useful?
The build date and time should probably be moved to the change log.
I'm thinking more of a version string that clearly identifies the version (or build number) and whether it's a 64 or 32 bit build.

[ultim]

@AVS+ devs
Any chance you could change the version string (from internal.h) to something more useful?
The build date and time should probably be moved to the change log.
I'm thinking more of a version string that clearly identifies the version (or build number) and whether it's a 64 or 32 bit build.

I'll add the architecture (x86/x64) to the version string, but for now I'd like to keep using dates as actual version numbers. The reason is, choosing any "traditional" numbering scheme makes little sense. Should we call it 0.x or 1.0? That would confuse people thinking it is older than 2.6. Should we call it 2.6? Not good because Avs+ is evolving much faster right now than classic Avs. 3.0? 4.0? Those have already been used in Avs. For anything larger, we didn't even have that many releases yet.

Of course one can argue that Avs+ is a totally separate project from Avs and thus Avs's historic numbering scheme shouldn't affect our versioning. You might even be right, but those who don't live on these forums (unlike us) might still mistake Avs+ for Avs and vice versa, at least initially. Also, even if, when do we increment the main version? Every time when a major feature is added (which according to the wish-list will be more than a few)? Or we can go the Linux way for libraries that only reaches 1.0 when the API is stable.

Of course all the above questions can be decided upon arbitrarily, but the different answers would make different amount of sense to different people, depending upon how you try to interpret the version numbers. IMHO using the release date as the version avoids such interpretation ambiguities, because if you see "2013-11-25" nobody will try to give it a special meaning, aside from the fact that it was released on that date. Which version is older or newer still remains clear, even compared to classic Avisynth. And as an added benefit, it also makes it easier to search for the corresponding release post on the forums, or for the appr. Git commit if tagging is ever forgotten.

[qyot27]

'4.0' is AvxSynth, not the original project (and I wouldn't really expect anyone to bring that into the discussion because I really don't know how widespread the knowledge that AvxSynth calls itself '4.0' is; even so, anyone who does know that probably also realizes that AvxSynth was based on 2.5.8 and probably considers the '4.0' to be irrelevant to the topic). And 3.0 never really went anywhere; it stalled and died in development hell before it ever became useful. All I really want to say is that my objection to '4.0' isn't because of it already being used by AvxSynth, it's because I think '4.0' is an excessively large jump.

A '3.1' would be enough to distinguish it from the old AVS3.0, IMO. Or it could go like the classic one did and jump from 2.0->2.5 - no one mistakes AviSynth 2.5 for 2.0. 3.5 wouldn't get mistaken for 3.0, but like '4.0', I still think that's a bit much to suddenly jump to. The angle that it's distinctly referred to as AviSynth+ would mean that the following version number would be attributed to the project name that came before it - AviSynth+ 3.1 =/= AviSynth 3.0 (for one, AVS3.0 was going to be implemented in Ruby). If it does increment with every new major feature, then it would quickly gain a much higher version number and there'd definitely be little chance of getting Classic and Plus mixed up. It could also be counted retroactively: 3.0 for the initial fork, 3.1 for 64-bit (maybe?), and so on. Git's tagging feature can come in handy whenever a change on this does happen (whatever it ends up being).


That kind of argument aside, I'd say sequential revision numbers are even more preferable than a date (not the build time displayed in the output of Version(), the use of the date in the revision), since you can build an earlier revision at a later date, but you can never build an older revision with a newer revision number (barring Git branch trickery; but for this purpose, let's assume the use of master). Git can generate this information with 'git rev-list --count branchname'. The X.X version can be used alongside it for 'stable' releases (like x264 or FFmpeg do), but the sequential revision is always incrementing upward and it's easy to tell how many commits separate any two revisions taken at random. If you want to isolate it to only those commits unique to AviSynth+, then have a 'trunk' branch that only consists of the commits from 2.6's CVS and use ^ notation to subtract them: 'git rev-list --count master ^origin/trunk'

[turbojet]

Why not keep it simple and version 1, 2, 3, etc. like x264/x265 does?

Also avisynth dll's have always been difficult to tell the difference in without naming it manually. This could easily change.

[Jeroi]

Why not keep it simple and version 1, 2, 3, etc. like x264/x265 does?

Also avisynth dll's have always been difficult to tell the difference in without naming it manually. This could easily change.

I second this. The 3.0 and 4.0 was ridicolous already because one was more like planned version of avs and second was linux port by some company.

I would like to see Avisynth+ start from 1 when it gets first stable release. Until that there could be 0.0.0 versions for development.

[qyot27]

Why not keep it simple and version 1, 2, 3, etc. like x264/x265 does?
You're not being clear.

x264 0.140.2377+29 8d4d435
(libswscale 2.5.101)
(libavformat 55.21.100)
(ffmpegsource 2.19.0.0)
built on Nov 11 2013, gcc: 4.8.2
configuration: --bit-depth=8 --chroma-format=all --disable-opencl
x264 license: Non-Free
libswscale/libavformat/ffmpegsource license: GPL version 3 or later
WARNING: This binary is unredistributable!
The version number in this case is 0.140, which is analogous to AviSynth(+) using '2.6' or whatever. The 2377+29 is the revision number, which is actually what I was just suggesting with the use of git rev-list --count. Under that model, avsplus would be:

Counting all revisions to the master branch, including those from classic AviSynth:
git rev-list --count master = r1545

Counting only the unique AviSynth+ revisions:
git rev-list --count master ^trunk = r292


Just to put something in perspective here, 292 revisions back from the 2.6 CVS trunk is commit efe6aa3f1b95647c2301df6e503622d8d51b4ba3, from Tue Jul 1 07:08:59 2008 +0000 - which if I'm not mistaken, was during 2.5.8's development cycle (2.5.8 was released at the end of 2008). There have been as many commits to AviSynth+ in the last 2½ months as there have been to classic AviSynth in the past 5½ years.

[qyot27]

But to add a different idea about versioning, how about something like this:

0.6.1.2.1545 (or 292)

this translates to:

0 = a quasi-standard non-1 major version, similar to x264
6 = the value of AVISYNTH_INTERFACE_VERSION (currently it's still 5, but there's been talk of bumping it to 6)
1 = the value of a hypothetical AVSPLUS_INTERFACE_VERSION
2 = MINOR_VERSION, for bugfixes or other non-API-breaking stuff, perhaps on some sort of monthly or commit-valued schedule
1545 (or 292) = REVISION_NUMBER based on total commits or only commits to AviSynth+

This way, the versioning reflects actual versions that developers need to take into consideration. One rather baffling thing about the advertised versions in classic AviSynth is that they don't match up with the value of AVISYNTH_INTERFACE_VERSION. '2' was pre-2.5.6 or 2.5.7 (can't remember), 3 was through 2.5.8, 4 was unused, 5 was the 2.6 alphas, and either 5 or 6 might be the final one for 2.6.

With the above scheme, some numbers are always incrementing (minor version and revision number), while others stay the same until something big happens and breaks the API, upon which the MINOR_VERSION is set back at zero (REVISION_NUMBER always climbs).

Of course, omitting the 0 and just using INTERFACE_VERSION for the MAJOR_VERSION would also make sense. Even though that'd mean versioning would be 5.whatever for AVISYNTH_INTERFACE_VERSION, at least the number would have a distinct meaning. If the rationale was just 'let's go higher than AvxSynth to avoid confusion' it'd be perpetuating that problem. If the rationale is to make the internal and external versions consistent, then I don't object.

[ryrynz]

The only problem I have with that is I think you'll forever be x.6.x.x.xxxx, there just aren't enough commits to Avisynth for it to be worth it basing Avisynth+'s version numbering off of it.
It should just have it's own version numbering system, this can happen whenever Ultim prefers to switch to it.

[ultim]

I'm still not convinced that any of these alternatives is better than dates (maybe with the exception that the build date does not necessarily equal the commit date). But there seems to be a consensus here and even on IRC that version numbers are better than dates (for whatever unnamed reason). Also the whole versioning "issue" is really not that important to me, so, fine, let's change it. But at least let's keep it simple and avoid "0.6.1.2.1545"-hell. If you're fine with it, I'd propose rXXXX-style sequence numbers (based on git rev-list --count master), tagging official releases x.x(.x)-style too.

[Groucho2004]

If you're fine with it, I'd propose rXXXX-style sequence numbers (based on git rev-list --count master), tagging official releases x.x(.x)-style too.
Works for me. :D

[qyot27]

rXXXX is fine by me - I used that notation in the CLI build guide alongside the output of the date command, and tend to find sequential revisions more informative than point versions most of the time.

My basic point was that a MAJOR.MINOR[.MICRO] type of point version should reflect *something* about the API directly, such as the interface version (if there is/are plans to have a discrete AVSPLUS_INTERFACE_VERSION, that would be the obvious choice).

[turbojet]

I had forgotten x264 had versions other than revision, always seen it referred to as revision number.

Is there any other filters besides resizers that use softwire? Or is there any other filters that may be slower than avisynth on an amd cpu?

Does a pull request for this help out? Or is 'issues' more appropriate?

[TurboPascal7]

Is there any other filters besides resizers that use softwire? Or is there any other filters that may be slower than avisynth on an amd cpu?

Filters that used softwire: Limiter, RGB<->YUV conversions (YV24, Y8, YUY2, BackToYUY2). Don't remember anything else.
Filters that may be slower on an amd cpu: no idea. No one tested.

Does a pull request for this help out? Or is 'issues' more appropriate?
Pull request is when you want to contribute e.g. some code to the project. Issues is more appropriate but there is no need. We played around with it a bit and decided that possible optimizations are not worth it right now. So yeah: for the time being, singlethreaded performance of resample filters will be worse in avs+ on some CPUs (this also affects sandy/ivy bridge afaik).

[turbojet]

Filters that used softwire: Limiter, RGB<->YUV conversions (YV24, Y8, YUY2, BackToYUY2). Don't remember anything else.
Filters that may be slower on an amd cpu: no idea. No one tested.

Would it help development to have these tested on AMD?

Pull request is when you want to contribute e.g. some code to the project. Issues is more appropriate but there is no need. We played around with it a bit and decided that possible optimizations are not worth it right now. So yeah: for the time being, singlethreaded performance of resample filters will be worse in avs+ on some CPUs (this also affects sandy/ivy bridge afaik).

Thanks for the explanation, I noticed you wrote pull requests were welcome in a reply about the slower performance but I wasn't sure if it was requested towards me.

[TurboPascal7]

Would it help development to have these tested on AMD?
Right now - not really. You're welcome to post your results and we might look into performance drops if there are some significant ones (which I really doubt because all other filters weren't nearly as optimized as resamplers) but we aren't specifically working on optimizing core filters right now since there is a lot of more important things to do.

[Groucho2004]

So yeah: for the time being, singlethreaded performance of resample filters will be worse in avs+ on some CPUs (this also affects sandy/ivy bridge afaik).
Did a quick test with some resizers - it's about 20% slower on my i5 2500K. Possibly worth mentioning, AVS+ uses more memory than the official Avisynth. With a script like this:
colorbars(width = 1920*2, height = 1080*2, pixel_type = "yv12").killaudio().assumefps(24000, 1001)
spline36resize(width() - 64, height() - 64)
spline64resize(width() + 64, height() + 64)
bicubicresize(width() - 64, height() - 64)
sincresize(width() + 64, height() + 64)

AVS+ uses ~150 MB, the official Avisynth ~ 55 MB.

[turbojet]

Right now - not really. You're welcome to post your results and we might look into performance drops if there are some significant ones (which I really doubt because all other filters weren't nearly as optimized as resamplers) but we aren't specifically working on optimizing core filters right now since there is a lot of more important things to do.

OK I'll pass on it for now.

Did a quick test with some resizers - it's about 20% slower on my i5 2500K. Possibly worth mentioning, AVS+ uses more memory than the official Avisynth. With a script like this:
colorbars(width = 1920*2, height = 1080*2, pixel_type = "yv12").killaudio().assumefps(24000, 1001)
spline36resize(width() - 64, height() - 64)
spline64resize(width() + 64, height() + 64)
bicubicresize(width() - 64, height() - 64)
sincresize(width() + 64, height() + 64)

AVS+ uses ~150 MB, the official Avisynth ~ 55 MB.

http://forum.doom9.org/showthread.php?p=1655396#post1655396 what were you benchmarking here? The numbers got me really optimistic.

[Groucho2004]

what were you benchmarking here? The numbers got me really optimistic.
Different script, different resolution, apples and oranges...

[turbojet]

So lanczos is faster but one or more of spline36, spline64, sinc, bicubic, sinc is slower?

[TurboPascal7]

So lanczos is faster but one or more of spline36, spline64, sinc, bicubic, sinc is slower?

It depends on the core size (different resizers), source and destination resolutions, your CPU model and the moon phase. I get 15% better performance on Groucho2004's script btw.

[innocenat]

I will explain problem with resizer slowdown in detail:

1. Larger memory footprint: Due to the new resizers are implemented as ResizeVertical.TurnRight.ResizeVertical.TurnLeft, this unfortunately requires two buffers to hold intermediate transposed frame. These buffers are quite large (frame size) and are the cause of increased memory footprint. But IMO, this is not really significant. If it bothers you, I can try to reduce memory footprint a bit, but it isn't possible to match original memory footprint with this implementation.

2. Performance varies between each resizers: Because without Softwire (a just-in-time assembler), the innermost loop of resizers cannot be unrolled to exact filter size. This loop runs a lot (~35 million times on Spline36Resize upscale from 720p to 1080p) and is very optimized. Loop overhead becomes noticeable here, and is reason why filters with large core size run more slower. Generally, Point > Bilinear > Bicubic = Spline16 > Spline36 = Lanczos3 > Spline64 = Lanczos.

3. Performance drop compared to original Avisynth: This is where things get really weird. All resizers run faster than original on Nehalem CPUs, but it look like they run slower on almost every other CPUs (including Intel's newer CPUs like Sandy Bridge). On older CPUs, it may caused by branch predictor cannot predict the innermost loop perfectly (the cycles require to run innermost loop is smaller than branch predictor penalties on most CPUs), but tests has confirmed that this is not a case for newer CPUs. There are some extra instruction generated by compiler in pre-innermost loop, which unfortunately is intrinsics limitation compared to pure asm.

tl;dr situation is weird and no one currently know what is actually happening.

[Groucho2004]

I will explain problem with resizer slowdown in detail:

1. Larger memory footprint: Due to the new resizers are implemented as ResizeVertical.TurnRight.ResizeVertical.TurnLeft, this unfortunately requires two buffers to hold intermediate transposed frame. These buffers are quite large (frame size) and are the cause of increased memory footprint. But IMO, this is not really significant. If it bothers you, I can try to reduce memory footprint a bit, but it isn't possible to match original memory footprint with this implementation.

2. Performance varies between each resizers: Because without Softwire (a just-in-time assembler), the innermost loop of resizers cannot be unrolled to exact filter size. This loop runs a lot (~35 million times on Spline36Resize upscale from 720p to 1080p) and is very optimized. Loop overhead becomes noticeable here, and is reason why filters with large core size run more slower. Generally, Point > Bilinear > Bicubic = Spline16 > Spline36 = Lanczos3 > Spline64 = Lanczos.

3. Performance drop compared to original Avisynth: This is where things get really weird. All resizers run faster than original on Nehalem CPUs, but it look like they run slower on almost every other CPUs (including Intel's newer CPUs like Sandy Bridge). On older CPUs, it may caused by branch predictor cannot predict the innermost loop perfectly (the cycles require to run innermost loop is smaller than branch predictor penalties on most CPUs), but tests has confirmed that this is not a case for newer CPUs. There are some extra instruction generated by compiler in pre-innermost loop, which unfortunately is intrinsics limitation compared to pure asm.

tl;dr situation is weird and no one currently know what is actually happening.
Actually, with "normal" scripts like simply scaling from 720p to 1080p or vice versa, the speed and memory usage differences are marginal. In some cases, AVS+ is even faster.

So, this is a rather insignificant issue. It's much more important that things work and people don't have to restart their process after 20 hours because Avisynth MT crashed for some reason.

[qyot27]

Not that I'm asking about its relevance to the resizer situation*, but if SoftWire was a JIT assembler, does that mean AviSynth used it for roughly the same reason that Schrödinger uses the Oil Runtime Compiler? The description for ORC seems rather similar to what it looks like SoftWire did.

*namely because it's stupid slow to build on anything not recent

EDIT: haha, apparently it got brought up a couple years ago in here (http://forum.doom9.org/showthread.php?t=162385).

[ultim]

A part of the increased memory consumption is also the result of the plugin manager. That is however just as planned, and the difference will depend on the amount of plugins (meaning it is constant, and will not increase with encoding time or script size/complexity).

As for some filters being best optimized currently for Nehalem, I might add the reason is because (as far as I've gathered but I'm not totally sure) all the plugin devs have Nehalem CPUs, so that is where they can test the most.

As for dynamic compilers, my personal opinion is that today it is best (most future proof) to go with LLVM. There is a related idea for Avs+ for GSoC, but not for the resizers. I wouldn't make the core dependent on *any* compiler library though, such functionality if anyone works on it will come as a separate plugin.

[jpsdr]

I've just installed/tested the actual release under Windows XP64 SP2.
The few tests i've made seems ok.
One thing to report about the installer, it doesn't create a menu.
Is it normal ?

Otherwise, thanks for your work at all the guys who are in this project.
Finaly, a properly x64 version is on the way.

[ultim]

I've just installed/tested the actual release under Windows XP64 SP2.
The few tests i've made seems ok.
One thing to report about the installer, it doesn't create a menu.
Is it normal ?

Otherwise, thanks for your work at all the guys who are in this project.
Finaly, a properly x64 version is on the way.

Thanks, in the name of everybody who worked / is working on the project. :D

About the Start Menu entries, yes they are not created by the current installer. Start Menu entries will come in a future release.

[zerowalker]

Does Avisynth+ have problems with using VirtualDub plugins?
I am getting errors here and there, though not sure if it´s Avisynth+ or not, haven´t tried using others yet.

Here are the errors, they come and go and it well, can be used in some way but it can also crash the decoder for the script (Avspmod often just produces errors, Virtualdub will crash if i give it the script 2 times in a row, and i think it fails to decode properly anyway).

Exception WindowsError: 'exception: access violation reading 0x0C412C00' in <bound method PIScriptEnvironment.__del__ of <avisynth.PIScriptEnvironment instance at 0x034DD828>> ignored

An out-of-bounds memory access (access violation) occurred in module 'VirtualDub'...
...reading address 069D2C00.


LoadVirtualDubPlugin("E:\Virtualdub\32 bit - Stable\plugins\addframenumber.vdf", "addframenumber",1)

That is the plugin i am using: http://www.neuron2.net/guest/

Well may be something totally else, and if so it´s off topic


EDIT:

Okay i got rid of it after testing all kinds of stuff, it had something to do with MixAudio being used after the VirtualPlugin, so probably just som incompatibility,
but i just changed the order and now it seems to work, made no sense having it after the Plugin anyway. So i guess it was just a False Alarm from my side.

[jpsdr]

I've installed a few days ago avisynth+ on Windows XP64 SP2 to test (and use) the x64 version.
The only functions i've used in my scripts for now are AviSource, WavSource, AudioDub, Spline36Resize, PointResize (which was not working on last avisynth64 release) and DGSource.
I've run several process of several hours with VDub64, no issue encounter.
I've run encodes of several hours with x264 (x64 version, of course, .avs file feed directly, but only AviSource used in that case), no issue encounter.
I've not tested a lot of avisynth functions, but at least, it seems to work properly with applications, and doesn't seem to have, at least for the few i've tested, any stability issue.
Good work, guys !
:thanks:

[aegisofrime]

Hi Guys,

I just tried Avisynth+ for the first time on my encoding rig, and I have been having some strange problems getting it to work.

I have the following script:

LoadPlugin("C:\DGIndex\DGDecode.dll")
DGDecode_mpeg2source("I:\Test\Test.d2v", info=3)
LoadPlugin("C:\Program Files (x86)\MeGUI\tools\avisynth_plugin\ColorMatrix.dll")
ColorMatrix(hints=true, threads=0)
QTGMC(Preset="Slower")
crop(4, 0, -4, 0)
#resize
Spline64Resize(720,480)
#denoise


Pretty simple QTGMC script. Now, the problem is that the script can be opened by some applications but not by others:

MeGUI: Stuck at opening script (preview window doesn't open), just keeps loading and loading. No errors thrown.

x264 input: Just says cannot open Test.avs

Virtualdub: Works :confused:

Lord Mulder's Simple x264 Launcher: Provides the most information of the bunch, says:

error: Cannot load file 'C:/DGIndex/DGDecode.dll'

Unless the loadplugin syntax was changed in Avisynth+, I have no idea what's wrong here. :confused:

Hope some of you can provide some wisdom here... Thanks!

[vivan]

Probably it's this issue: http://forum.doom9.org/showthread.php?p=1655169#post1655169

[real.finder]

Hi :)

I have a suggestion

Function to change the path of autoload folder temporarily in the script


something like:-

PluginsDir("new plugins path will be here")

and for + Plugins

PluginsPlusDir("new plugins+ path will be here")


in the top of script, like:-

PluginsDir("D:\plugins")
PluginsPlusDir("D:\plugins+")
MPEG2Source("C:\Documents and Settings\user\Desktop\video.d2v")
Tfm()

Would be a wonderful thing for users who without authority of admin :)

--------


On the other hand, I'm still waiting for MT support in order to use avs+ mainly in my encoding


thanks :)

[innocenat]

PluginsDir("new plugins path will be here")

and for + Plugins

PluginsPlusDir("new plugins+ path will be here")


try AddAutoloadDir("C:\path\to\dir")

You don't need separate folder --- the command only works in Avs+ which can load either anyway,

[real.finder]

try AddAutoloadDir("C:\path\to\dir")

You don't need separate folder --- the command only works in Avs+ which can load either anyway,

aha, so it's already exist :)

find it here (http://forum.doom9.org/showpost.php?p=1646315&postcount=67)

thank you

[aegisofrime]

Probably it's this issue: http://forum.doom9.org/showthread.php?p=1655169#post1655169

Thanks for the reply, however it doesn't seem to be the case since the script works in Virtualdub, and anyway my LoadPlugin is before the DGDecode call... :confused:

[StainlessS]

@aegisofrime, does Info() at end of script show anything of interest in VD ?
NOTE, VD has it's own decoders/colorspace converters.

EDIT: Empty plugins directory temporarily (except for any necessary plugs).
EDIT: Although problem seems directory related for DGDecode.
EDIT: You could verify DGIndex load as cause using below


RT_Debug("Loading DGIndex")
LoadPlugin("C:\DGIndex\DGDecode.dll")
RT_Debug("DGIndex Loaded")


And view output in DebugView (google) (RT_Debug requires RT_Stats plug).

[aegisofrime]

@aegisofrime, does Info() at end of script show anything of interest in VD ?
NOTE, VD has it's own decoders/colorspace converters.

EDIT: Empty plugins directory temporarily (except for any necessary plugs).
EDIT: Although problem seems directory related for DGDecode.
EDIT: You could verify DGIndex load as cause using below


RT_Debug("Loading DGIndex")
LoadPlugin("C:\DGIndex\DGDecode.dll")
RT_Debug("DGIndex Loaded")


And view output in DebugView (google) (RT_Debug requires RT_Stats plug).

I took your advice of cleaning out my plugins folder, and moved back the important ones for QTGMC. It works now, so it seems there was an offending filter in my admittedly messy plugins folder. The strange thing is that the same plugin collection works on my Windows 8 laptop without a hitch. :confused:

If I have time I will track down and see what's the filter that's causing this issue.

[StainlessS]

I had some problems some time ago with Avisynth ffmpegSource() and Vdub Plugins32 FFImputDriver.vdplugin
and the ffdlls directory in VDub plugin32, the presence of ffdlls in vd plugs dir caused some kind of conflict
with avisynth, cant remember exactly what the problem was, but seemed related to ffmpegSource and/or
FFImputDriver versions.
Might want to check out by restoring original plugins but without the suspect plugins in AVS/VD plugs dir.
I think the problem was something to do with directory search order being changed, it may have nothing to do with the
problem, but would be first thing I would try out.

[qyot27]

I had some problems some time ago with Avisynth ffmpegSource() and Vdub Plugins32 FFImputDriver.vdplugin
and the ffdlls directory in VDub plugin32, the presence of ffdlls in vd plugs dir caused some kind of conflict
with avisynth, cant remember exactly what the problem was, but seemed related to ffmpegSource and/or
FFImputDriver versions.
Might want to check out by restoring original plugins but without the suspect plugins in AVS/VD plugs dir.
I think the problem was something to do with directory search order being changed, it may have nothing to do with the
problem, but would be first thing I would try out.
That would have to do with the really really early builds of FFmpegSource 1.x that relied on shared versions of the FFmpeg libraries. The builds switched to including those statically somewhere in the middle of the 1.x dev cycle (back in like, 2008 or something) and it's never been done as shared for Windows builds of FFMS2.

[ultim]

A second bugfix release is available. Besides proudly wielding the version number "0.1 (r1555)", the most important changes are:

A fix for the autoload issue reported here (http://forum.doom9.org/showthread.php?p=1655169#post1655169).
A fix for TemporalSoften which potentially resulted in crash.
A fix for some filters not loading under specific circumstances. Discovered on WriteFileStart.
A fix for the "return" script statement not returning from the current function if used inside if/while/for etc.

This release is a nice opportunity for you to try out AviSynth+ if you didn't already. Unless some major issue pops up, the next release will bring larger changes.

There is a slight change in behavior in r1555, made necessary by the fix for the autoload issue. Previously, plugin autoloading started automatically if forced by the AutoloadPlugins() function, or if an unknown(=external) function was found. Beginning with this release there is also a third condition, autoloading will also happen if any LoadPlugin() is issued, and it will happen right before the LoadPlugin() is executed. This was necessary to preserve compatibility with scripts for classic AviSynth.

Anyway, enjoy our latest and greatest release. Download (http://www.avs-plus.net/) from the homepage.

[ultim]

In other news, there is a GitHub organization (https://github.com/AviSynth) now for AviSynth. If you're the developer of an open source plugin, script or utility related to AviSynth, join the organization so that all avs-related code can be found in one place. Of course you still preserve full admin privileges over your repository upon joining, but visitors will be able to find all AviSynth related projects at a glance, which is good for both your code, for AviSynth, and for the visitor/community.

If you're already a GitHub user, joining is easy, you just need to tell me or TurboPascal7 that you'd like to join, and let us know your GitHub username. If you're not using GitHub already, I strongly recommend you create an account. Not (only) because of the (1) organization, but because (2) it will greatly simplify accepting contributions from others (be it code or feedback), and (3) GitHub will also make it much easier for you to host your code and binary releases.

So, fellow Avs Devs, join the AviSynth organization on GitHub (https://github.com/AviSynth).

[Gavino]

A fix for the "return" script statement not returning from the current function if used inside if/while/for etc.
I can understand why you might consider the old behaviour to be a bug, but I made GScript work that way because that was what already happened with a 'return' inside a try/catch block (exit only from current block). With this change, Avisynth+ is now inconsistent with Avisynth in this respect.

[ultim]

I can understand why you might consider the old behaviour to be a bug, but I made GScript work that way because that was what already happened with a 'return' inside a try/catch block (exit only from current block). With this change, Avisynth+ is now inconsistent with Avisynth in this respect.

We acknowlege this small inconsistency, but we believe that there almost no scripts affected, and so the advantages greatly outweight the downsides of this small deviation from AviSynth.

[Mystery Keeper]

Since you now let user customize plugins path, would you please add separate path for .avsi scripts?

[TurboPascal7]

Since you now let user customize plugins path, would you please add separate path for .avsi scripts?
What for?

[Mystery Keeper]

To not copy them across 32 and 64 bit folders and keep them separated from the binaries.

[real.finder]

hi, what AutoloadPlugins() function do?

Did not mention here (http://forum.doom9.org/showpost.php?p=1646315&postcount=67)

[ultim]

hi, what AutoloadPlugins() function do?

Did not mention here (http://forum.doom9.org/showpost.php?p=1646315&postcount=67)

It simply forces autoloading at the point it is called. Not usefull for scripts, it was meant to support editors like AvsPmod.

[Groucho2004]

I have been testing the 32 bit AVS+ version on XP64 with the following script to drive the memory usage above 2G:
setmemorymax(16384)
n = 9000
colorbars(width = n * 2, height = n, pixel_type = "yv12").killaudio().assumefps(24000, 1001)
trim(0,19)
pointresize(width() - 64, height() - 64).turnleft()
pointresize(width() + 64, height() + 64).turnright()
a=selectevery(3, 0).addborders(0,0, 16,0).crop(0,0, -16,0)
b=selectevery(3, 1).addborders(0,0, 32,0).crop(0,0, -32,0)
c=selectevery(3, 2).addborders(0,0, 64,0).crop(0,0, -64,0)
interleave(a, b, c)

Results with the "normal" Avisynth 2.6 Alpha5:
Frames processed: 20 (0 - 19)
FPS (min | max | average): 0.24 | 0.25 | 0.25
CPU usage (average): 25%
Thread count: 2
Physical Memory usage (peak): 2330 MB
Virtual Memory usage (peak): 2324 MB
Time (elapsed): 000:01:21.186


And with the latest AVS+ (x86):
Frames processed: 20 (0 - 19)
FPS (min | max | average): 0.53 | 0.61 | 0.60
CPU usage (average): 25%
Thread count: 2
Physical Memory usage (peak): 3030 MB
Virtual Memory usage (peak): 3022 MB
Time (elapsed): 000:00:33.140

AVS+ is obviously much faster but also uses a lot more memory. This might be an isolated case but I thought it might be worth reporting.

[ultim]

What eats memory are the resize functions. If you remove them, memory consumption sinks drastically.

setmemorymax(16384)
n = 9000
colorbars(width = n * 2, height = n, pixel_type = "yv12").killaudio().assumefps(24000, 1001)
trim(0,19)
turnleft()
turnright()
a=selectevery(3, 0).addborders(0,0, 16,0).crop(0,0, -16,0)
b=selectevery(3, 1).addborders(0,0, 32,0).crop(0,0, -32,0)
c=selectevery(3, 2).addborders(0,0, 64,0).crop(0,0, -64,0)
interleave(a, b, c)

Frames processed: 20 (0 - 19)
FPS (min | max | average): 1.38 | 1.65 | 1.59
CPU usage (average): 12%
Thread count: 4
Physical Memory usage (peak): 1201 MB
Virtual Memory usage (peak): 1214 MB
Time (elapsed): 000:00:12.605

[Gavino]

We acknowlege this small inconsistency, but we believe that there almost no scripts affected, and so the advantages greatly outweight the downsides of this small deviation from AviSynth.
I agree the new semantics are more useful.
Perhaps IanB could be persuaded that the way Avisynth currently handles 'return' in try/catch is a bug, and should be fixed.

Looking at the changes to the Avisynth+ source code, I believe there is a flaw in the way 'return' has been implemented.
You have put the code to handle the new ReturnExprException into ScriptEnvironment::Invoke(). This will catch a 'return' from both script level and from a user function, but not from run-time scripts (eg in ScriptClip()), since the run-time filters call the parser directly and do not use Invoke(). I haven't actually tried it, but I think you will now get an unhandled exception if you use 'return' in a run-time script.

Conceptually, Invoke() also feels like the wrong place to handle this, as this exception is really just a parser thing and it would be cleaner to handle it fully inside the parser code.

My preferred solution would be to introduce a new Expression subclass 'ExpRoot' which the parser would use for the root of the expression tree (in ScriptParser::Parse()) and for function bodies (in ScriptParser::ParseFunctionDefinition). The ReturnExprException could then be caught in the Evaluate() of this new subclass.

[ultim]

I agree the new semantics are more useful.
Perhaps IanB could be persuaded that the way Avisynth currently handles 'return' in try/catch is a bug, and should be fixed.

Looking at the changes to the Avisynth+ source code, I believe there is a flaw in the way 'return' has been implemented.
You have put the code to handle the new ReturnExprException into ScriptEnvironment::Invoke(). This will catch a 'return' from both script level and from a user function, but not from run-time scripts (eg in ScriptClip()), since the run-time filters call the parser directly and do not use Invoke(). I haven't actually tried it, but I think you will now get an unhandled exception if you use 'return' in a run-time script.

Conceptually, Invoke() also feels like the wrong place to handle this, as this exception is really just a parser thing and it would be cleaner to handle it fully inside the parser code.

My preferred solution would be to introduce a new Expression subclass 'ExpRoot' which the parser would use for the root of the expression tree (in ScriptParser::Parse()) and for function bodies (in ScriptParser::ParseFunctionDefinition). The ReturnExprException could then be caught in the Evaluate() of this new subclass.

Correct. I will hotfix it tonight and edit this post when done. Thank you!
EDIT: The release is up. Contains a fix for the issue pointed out by Gavino, and also an installer fix as well as some visual update.

[jpsdr]

What eats memory are the resize functions.


Maybe in that case the question should be "Why...?".
Why the resize functions in avisynth+ use more memory ?

[TurboPascal7]

Maybe in that case the question should be "Why...?".
Why the resize functions in avisynth+ use more memory ?

I described it here (http://forum.doom9.org/showpost.php?p=1655121&postcount=300) in part 9.
Now, current implementation is less memory efficient than it could be and this will get fixed some time soon. But we won't go back to the old way of doing it so avs+ resizers will always eat more memory than original.

[jpsdr]

Ok, thanks for the information.

[hector40]

Hi, this script crash avspmod.

BlankClip(length=1000, width=720, height=480, pixel_type="yv12", fps=23.976)
a=last
crop(718, 0, -0, -0)
Repair(blur(1.0).AddBorders(718,0,0,0), last.AddBorders(718,0,0,0),1)
crop(718, 0, -0, -0)
overlay(a, last, mode="blend", opacity=1.0, x=718)

[wOxxOm]

Hi, this script crash avspmod.yeah it does crash avs+ here too, so I simplified the script, the essential code is: BlankClip(1000,720,480,"yv12")
overlay(last)
callstack:> avisynth.dll!convert_yv24_chroma_to_yv12_sse2(unsigned char * dstp, const unsigned char * srcp, int dst_pitch, int src_pitch, int dst_width, const int dst_height) Line 309 C++
avisynth.dll!Convert444ToYV12::ConvertImage(Image444 * src, PVideoFrame dst, IScriptEnvironment * env) Line 410 C++
avisynth.dll!Overlay::GetFrame(int n, IScriptEnvironment * env) Line 277 C++
avisynth.dll!Cache::childGetFrame(int n, IScriptEnvironment * env) Line 356 C++
avisynth.dll!Cache::GetFrame(int n, IScriptEnvironment * env) Line 557 C++
avisynth.dll!avs_get_frame(AVS_Clip * p, int n) Line 170 C++

On this line (https://github.com/AviSynth/AviSynthPlus/blob/master/avs_core/filters/overlay/444convert.cpp?source=c#L309) dst_width==360 for the chroma plane which is not mod16, so _mm_store_si128 crashes.
Should be an unaligned store which is also used in other functions:
_mm_storeu_si128(reinterpret_cast<__m128i*>(dstp+dst_width-16), avg);

[zerowalker]

Is it possible to add Wave64 (or rather Wave over 4gb) support?

It´s now possible to open it, but the time will incorrect, it will stop at some point before the actual end (Probably when it reaches the 4gb limit).

There is a workaround to use plugins, but most of the time Directshowsource does the work best.

[real.finder]

hi

About MT and setmtmode and the old closed Source filters

It would be better if setmtmode applied only on those filters

Because there are some filters will be slow with setmtmode, no matter mode were used

will be slower than without setmtmode, such as ffms2, especially with large files

So it would be good to apply setmtmode automatically with the proper mode on those filters only

And after one filter with setmtmode enabled, if there a filter with a different setmtmode mode will be changed automatically to that mode

in this case, avs+ will need a particular directory for those filters that will be used with setmtmode and the proper mode for each one of them

Otherwise, setmtmode will be cancel by the default

But the problem is the cancellation, setmtmode 5 and setmtmode 6 not good for this

Because I tried them with ffms2 and the speed was reduced, and of course I can use mp_pipeline to overcome such a problems, but it better to have a mechanism to stops setmtmode in the same script

----


For open source and the modern filters, they will use an internal MT

or running setmtmode with the proper mode into avs script, and cancel it if the next filter not need setmtmode

or it better to be a new and safe mt between the avs and the filters to be used in those cases away from setmtmode


thanks :)

[wOxxOm]

real.finder, instead of the outdated ffms2 can you please test L-SMASH Source (http://forum.doom9.org/showthread.php?t=167435)'s LSMASHVideoSource (for mp4/mov) and LWLibavVideoSource (for everything) in your avs mt workflow? If it isn't affected then maybe there is no problem to solve, just switch to L-S...

[real.finder]

real.finder, instead of the outdated ffms2 can you please test L-SMASH Source (http://forum.doom9.org/showthread.php?t=167435)'s LSMASHVideoSource (for mp4/mov) and LWLibavVideoSource (for everything) in your avs mt workflow? If it isn't affected then maybe there is no problem to solve, just switch to L-S...

I mentioned ffms2 for example only

There may be other filters do not like setmtmode, and this known thing

in the past, I tested put setmtmode before mp_pipeline and make the process slower too

[jpsdr]

I've quickly tested QTGMC v3.33 (just try to open file and view a few frames) under Windows XP64 SP2, with r1555.

Seems to works fine with VDub x86.

Not working with VDub x64, error message :

Cache Filter returned invalid response to CACHE_GETCHILD_CACHE_MODE. 78312696
.../plugins64/QTGMC.avsi line 784
.../plugins64/QTGMC.avsi line 387
.../Deinterlace_v2.avs line 2


Line 784 :
diff = mt_makediff( Ref, Input, U=3,V=3 )
Line 386 :
repair0 = (IsClip(srchClip) || Rep0 == 0) ? binomial0 : binomial0.QTGMC_KeepOnlyBobShimmerFixes( bobbed, Rep0, (RepChroma && ChromaMotion) )

My script :

AVISource("File.avi")
QTGMC(EdiMode="nnedi3",ChromaEdi="nnedi3",blocksize=8,overlap=4,search=5,searchparam=4,EdiThreads=6,ChromaMotion=false)
SelectEven()


Of course, this was working with avisynth64_4-16-10.

[real.finder]

you need mt_masktools-26.dll compile with new avs interface

Available in x86 by now, not for x64 until now, At least I didn't see one for 64 until now

Many filters need to have 64 ver.

[jpsdr]

I have the x64 version of v2.0a48 for avs 2.6... Another new one is required ?

[TurboPascal7]

I have the x64 version of v2.0a48 for avs 2.6... Another new one is required ?

Yes. I will publish a fork of masktools some time soon.
It doesn't mean that you'll be able to safely run QTGMC though since there are many more filters used.

[real.finder]

I have the x64 version of v2.0a48 for avs 2.6... Another new one is required ?

avs 2.6 interface changed in alpha 4, So it need to recompiling with it, there is x86 by now only (http://tmod.nmm-hd.org/Misc/mt_masktools-26-for-2.6alpha4.7z) as I said

official mt_masktools-26.dll will not work with avs 2.6 alpha 4 and up

Of course avs+ also

you can use mt_masktools-25.dll but make sure there is no other mt_masktools in plugins autoload folder dir

[jpsdr]

I've try to compile the project (VS2010) (source taken from v2.0a48)... Without succes. I've installed yasm according the description in the readme (vysasm.exe in "../VC/Bin" and props/target/xml in "../v4.0/BuildCustomizations").

When i try to generate project, i've the following error :

1>------ Début de la génération*: Projet*: common, Configuration*: release-avs26 Win32 ------
1>F:\PRG\Visual_2010\masktools-v2.0a48\common\build\yasm.targets(73,5): error MSB4023: Impossible d'évaluer la métadonnée d'élément "%(Extension)". Impossible d'appliquer la métadonnée d'élément "%(Extension)" au chemin d'accès ""Win32\release-avs26\\_functions.obj"". Caractères non conformes dans le chemin d'accès.
2>------ Début de la génération*: Projet*: masktools, Configuration*: release-avs26 Win32 ------
2>F:\PRG\Visual_2010\masktools-v2.0a48\masktools\build\yasm.targets(73,5): error MSB4023: Impossible d'évaluer la métadonnée d'élément "%(Extension)". Impossible d'appliquer la métadonnée d'élément "%(Extension)" au chemin d'accès ""Win32\release-avs26\\_functions.obj"". Caractères non conformes dans le chemin d'accès.
========== Génération*: 0 a réussi, 2 a échoué, 0 mis à jour, 0 a été ignoré ==========

Translation is something like :

1>------ Start generation*: Project*: common, Configuration*: release-avs26 Win32 ------
1>F:\PRG\Visual_2010\masktools-v2.0a48\common\build\yasm.targets(73,5): error MSB4023: Impossible to evaluate the metadata "%(Extension)". Impossible to apply the metadata "%(Extension)" to acces path ""Win32\release-avs26\\_functions.obj"". Characters not conform in the acces path.
2>------ Start generation*: Projet*: masktools, Configuration*: release-avs26 Win32 ------
2>F:\PRG\Visual_2010\masktools-v2.0a48\masktools\build\yasm.targets(73,5): error MSB4023: Impossible to evaluate the metadata "%(Extension)". Impossible to apply the metadata "%(Extension)" to acces path ""Win32\release-avs26\\_functions.obj"". Characters not conform in the acces path.
========== Génération*: 0 success, 2 failed, 0 updated, 0 ignored ==========

[Groucho2004]

I've try to compile the project (VS2010) (source taken from v2.0a48)... Without succes.
How is this related to AVS+?

[jpsdr]

Because... ... euh... after being able to compile the original source, i wanted to try to compile with the new avisynth.h from AVS+ ? ... not good...?
Ok sorry...

[TurboPascal7]

As I told you I will publish a fork of masktools some time soon. If you want to build it right now for whatever reason, feel free (https://github.com/tp7/masktools). But you'll need VS2012 for that.

[jpsdr]

Ok, thanks.

[ultim]

hi

About MT and setmtmode and the old closed Source filters

It would be better if setmtmode applied only on those filters

Because there are some filters will be slow with setmtmode, no matter mode were used

will be slower than without setmtmode, such as ffms2, especially with large files

So it would be good to apply setmtmode automatically with the proper mode on those filters only

And after one filter with setmtmode enabled, if there a filter with a different setmtmode mode will be changed automatically to that mode

in this case, avs+ will need a particular directory for those filters that will be used with setmtmode and the proper mode for each one of them

Otherwise, setmtmode will be cancel by the default

But the problem is the cancellation, setmtmode 5 and setmtmode 6 not good for this

Because I tried them with ffms2 and the speed was reduced, and of course I can use mp_pipeline to overcome such a problems, but it better to have a mechanism to stops setmtmode in the same script

----


For open source and the modern filters, they will use an internal MT

or running setmtmode with the proper mode into avs script, and cancel it if the next filter not need setmtmode

or it better to be a new and safe mt between the avs and the filters to be used in those cases away from setmtmode


thanks :)

In AVS+, SetMTMode (with a different name) is implemented similarly to your idea, with minor differences. The way it works is instead of specifying the MT mode for all the filters that follow, AVS+'s setmtmode accepts the filter's name as its first parameter, and it then applies to all instances of that filter, and only to that filter. So basically it is what you have said, in that you can set the mt-mode for each filter once, and AVS+ will switch between mt modes automatically.

This has a couple of nice advantages:

First, obviously, your scripts don't need to be modified in their middle. There is no need to insert dozens of setmtmode calls all over a script, and you can keep the actual script logic and setmtmode calls separate cleanly.
Second, this approach allows you to move the setmtmode calls for all of your installed plugins into one script that contains nothing but all the setmtmode calls, which you can then include in your other scripts. Meaning you'll only need to maintain one set of setmtmode calls.
Even better than my previous point, you don't even need to include the setmtmode specifications in all of your scripts, you can just let it autoload like any other .avsi script. In AVS+, setmtmode is not what enables parallel processing, so you can safely call setmtmode in any script even if you don't want to run it multithreaded. Multithreading is enabled by a different call that you can add to only those scripts that use MT-capable plugins. (setmtmode merely specifies how to multithread a filter if MT is enabled at all)


Miscellaneous notes:

In AVS+, "SetMTMode" is actually called "SetFilter(MT)Mode". This is partially to differentiate scripts written for AVS+ from scripts written for AVS-MT, and also to more clearly indicate that this call does not set the MT mode globally, but only for a particular filter.
However, the same call can also be used to set the default MT mode for all filters that have no other MT mode set.
Plugins can also indicate to AVS+ which MT mode they need over their API. If this is specified by a plugin, it will normally take precedence over the mode set by the script, because it is more likely to be correct (a user's manual setting can easily be out-of-date, or just simply incorrect). If a plugin specifies the MT mode for itself, there is no need to specify it in scripts at all.
SetFilterMTMode also has a second, boolean "force" parameter, optional and defaulting to false. If this is set to true in a script, that mode will take precedence over any other calls for the same filter where "force" was false. It will then also take precedence over the mode specified by the plugin internally.


The priority scheme of setfiltermode calls may sound complex, but it is really simple. To summarize:

If a filter has a forced mode set, it will use that.
Otherwise, it will use a non-forced mode, if it has any.
If it does not have any mode set, it will use the default mode for all filters.
Mode setting does not have any effect until MT is actually enabled (so it is still safe to set even the default mode in any script).


---------------------------

What I have described above is already done and implemented, except for filters being able to indicate their MT mode on their own to AVS+. However, due to some other reasons related mostly to caching, the MT-version of AVS+ isn't ripe enough yet, which is why it isn't included in the public builds currently.

[ultim]

Cache Filter returned invalid response to CACHE_GETCHILD_CACHE_MODE.

For future reference, if anybody sees the message above, it means the plugin is not compatible with AviSynth 2.6a4. For clarification, Avs+ uses the 2.6a4 interface.

What you can do at this point, is look in the forums if anybody has published an updated build of the same plugin, or recompile the plugin yourself.

Recompiling a 2.5 plugin for 2.6a4 will need some modifications to the plugin's source, but only very simple things. If you're a developer, it will definitely be easy (given that the pre-2.6a4 version can be recompiled, which for some plugins will be much harder than the actual porting to 2.6a4).

[ryrynz]

Looking forward to seeing MT in action, will it follow the same logic and performance as SEt's MTmodes?
If there is possibly no need to specify modes in scripts at all if the filter overrides it, will you provide a message stating this on runtime so it can be removed by the user? I'd rather not have unecessary lines in the script.
I was hoping to see a MT build soon, probably next month now?

[real.finder]

The only thing that makes me use SEt's build in my encoding by now and don't use avs+ is mt

So I also wait avs+ MT build

[ultim]

Looking forward to seeing MT in action, will it follow the same logic and performance as SEt's MTmodes?
Not sure what you mean by "same logic". It will have the same MT modes 1-3 as SEt's version.


If there is possibly no need to specify modes in scripts at all if the filter overrides it, will you provide a message stating this on runtime so it can be removed by the user? I'd rather not have unecessary lines in the script.

Maybe, in the future, but not right at the start. Those "unnecessary" lines should not disturb you as they will not have any effect, and if you autoload the setfiltermode lines they won't pollute your scripts.


I was hoping to see a MT build soon, probably next month now?
I was planning originally to release in november, but RL stepped in to prevent that. So no guarantees, but I do hope I'll get more opportunities to work on it in the holiday season. Rest assured, MT is coming, and more features will after that.

[jpsdr]

I have somehow just a little question.
I'm trying to port to x64 nnedi3 v0.9.4. My first step is to put on external asm files all the _asm inside. I've just begun (here (https://github.com/jpsdr/NNEDI3)), it's going slowly, but i'm progressing. When done and tested, my second step will be to convert the asm file to x64. But, for now, during the first step, i'm begining to think of the second step, and i've noticed something different from what i'm used to in VDub (i've made personnaly several filters for VDub, but asm was always in asm files, never inline, so updating to x64 was less less difficult).
In VDub, all pointer related data (offset/pitch) are in ptrdiff_t, which allow an excellent portability x86/x64, specialy when you write asm functions, and pass pitch parameters which are used directly to compute on address data.
I've noticed, in the code of nnedi3, and after taking a look at avisynth.h, that in avisynth these parameters are int. Int is 32 bits even on x64... In the process of updatind and making avisynth more portable for x64, don't you think it could be interesting to think about having offset/pitch pointer parameters of avisynth picture data in ptrdiff_t ? It's the same thing as int on x86 (32 bits signed), but they will be 64 bits signed on x64, more logical for pointer offset data.
Well, it's just my little thought.

[ultim]

You are not the first to suggest that some data types should be changed, but we cannot do so without breaking existing x64 filters.

Either way, offset/pitch being only ints isn't really a problem. Offset merely stores the displacement of a frame's plane from it's base pointer, so this is not a problem since your planes will be lot less smaller than 2GBs. Similarly, pitch is the displacement from the beginning from one line to the next, and since a single row of samples will rarely hit 2GBs in size, this doesn't pose any realistic limitation. So offset/pitch aren't pointers, both are offsets from pointers within a single frame, hence I don't see how this is a portability issue. It won't even cause a problem with pointer arithmetic in C. You might need to take special precautions in ASM to get correct results out of pointer math, but I'm not the expert in that area.

[TurboPascal7]

If you plan on going external asm, please consider using x264 abstraction thingy over simply changing the code to be x64-only. It's always better to have a single source that can compile to both x64 and x86 instead of two different codebases. You can find example usage in x264 or vapoursynth.

[jpsdr]

Typical could be :
In .cpp file :

extern "C" void fn(void *ptr,ptrdiff_t pitch);


x86 asm file :

fn proc ptr:dword,pitch:dword

public fn

push ebx

mov ebx,ptr
mov ecx,pitch.
mov eax,dword ptr[ebx+ecx]
.....


x64 asm file :

fn proc public frame

push rbx
.pushreg rbx
.endprolog

mov rbx,rcx
mov rcx,rdx
mov eax,dword ptr[rbx+rcx]
....

Note : In x64 the 3 first parameters are passed in registers rcx (ptr) and rdx (pitch) and r8d but not here because there is only 2 parameters.

Will work without issue for positive and negative pitch in both x86 and x64.

If you have this instead :
In .cpp file :

extern "C" void fn(void *ptr,int pitch);


x86 asm file :

fn proc ptr:dword,pitch:dword

public fn

push ebx

mov ebx,ptr
mov ecx,pitch.
mov eax,dword ptr[ebx+ecx]
.....

No change, still working.

x64 asm file :

fn proc public frame

push rbx
.pushreg rbx
.endprolog

mov rbx,rcx
xor rcx,rcx
mov ecx,edx
mov eax,dword ptr[rbx+rcx]
....

Will not work on negative pitch !!!

Issue can be with negative value in asm functions. If you pass in x64 mode to an asm function a pointer (on 64 bits) and a pitch but on 32 bits for it, it will work only on positive value.
In VDub pitch can be negative. If in avisynth pitch is always positive, issues may not occurs.

[jpsdr]

If you plan on going external...
My purpose is to try to do the same thing i've already done with my VDub filters : One cpp file working for both x86 and x64, 2 asm files (one x86 and one x64).
Will i success ? That's another story.

[TurboPascal7]

My purpose is to try to do the same thing i've already done with my VDub filters : One cpp file working for both x86 and x64, 2 asm files (one x86 and one x64).
Will i success ? That's another story.

Why two asm files when you can do this in one, simply abstracting register names and maybe some other things?
But whatever works for you, I guess.

[foxyshadis]

Some x64 asm can definitely benefit from the extra registers, but so few that the maintenance cost of two asm functions usually outweighs it. Even more so when you start optimizing for various architectures (SSE2, SSSE3, AVX) and 16-bit depth. x86inc is SO useful, and x86util does handy work papering over some more differences. You really should consider using them, you might be surprised.

[ajp_anton]

Could you make so imagesource returns the original colorspace, e.g. jpeg images encoded in 4:2:0 or 4:4:4, instead of converting them to RGB?
Perhaps with the argument pixel_type="original", if you want to keep it compatible with old scripts that assume RGB.

[ultim]

x64 asm file :

fn proc public frame

push rbx
.pushreg rbx
.endprolog

mov rbx,rcx
xor rcx,rcx
mov ecx,edx
mov eax,dword ptr[rbx+rcx]
....

Will not work on negative pitch !!!


You could just sign extend your edx into rcx to solve this "problem". For example (untested):

fn proc public frame

push rbx
.pushreg rbx
.endprolog

mov rbx,rcx
movsxd rcx,edx
mov eax,dword ptr[rbx+rcx]
....

This should work for x64 with both positive and negative pitches, and even requires less assembly lines.

[ultim]

Could you make so imagesource returns the original colorspace, e.g. jpeg images encoded in 4:2:0 or 4:4:4, instead of converting them to RGB?
Perhaps with the argument pixel_type="original", if you want to keep it compatible with old scripts that assume RGB.

The current image library (DevIL) in use by AviSynth doesn't allow that, so no. Neither it is possible with many other generic image decoder libraries, as still images are usually composed on RGB surfaces. If anybody knows a still image library that can do that, I'd be happy to have a closer look at it. On the other hand, if it is only possible with format-specific decoders, then I'd only have it as a separate external plugin.

[qyot27]

FFMS2 can be used for stills (see the FFImageSource function in FFMS2.avsi), and it returns the original's colorspace. I just tested it, and it returned 4:2:0 and 4:2:2 directly.

[ultim]

FFMS2 can be used for stills (see the FFImageSource function in FFMS2.avsi), and it returns the original's colorspace. I just tested it, and it returned 4:2:0 and 4:2:2 directly.

Does it really return the original image data unchanged? Or maybe it gets RGB data internally from the format's lib, then ffmpeg converts back to planar on its own?

[jackoneill]

My purpose is to try to do the same thing i've already done with my VDub filters : One cpp file working for both x86 and x64, 2 asm files (one x86 and one x64).
Will i success ? That's another story.

I already converted nnedi3's asm into an external file that you can assemble with yasm. It works for both 32 and 64 bit. If Avisynth(+) doesn't allow negative strides, you can probably use it without any changes.

https://github.com/dubhater/vapoursynth-nnedi3

[ultim]

I already converted nnedi3's asm into an external file that you can assemble with yasm. It works for both 32 and 64 bit. If Avisynth(+) doesn't allow negative strides, you can probably use it without any changes.

https://github.com/dubhater/vapoursynth-nnedi3

It is extremely rare, but I do think negative pitches exist in Avisynth, for example when some plugin tries to optimize vertical flips away (I faintly remember seeing it somewhere). Anyway, the workaround is really simple, just sign-extend like I have shown a few posts earlier, and it will work for negative pitches too.

[Reel.Deel]

...If anybody knows a still image library that can do that, I'd be happy to have a closer look at it. On the other hand, if it is only possible with format-specific decoders, then I'd only have it as a separate external plugin.

vsimagereader (https://github.com/chikuzen/vsimagereader) it's able to decode JPEG without converting to RGB. All of the libraries used are listed at the bottom.

[ultim]

vsimagereader (https://github.com/chikuzen/vsimagereader) it's able to decode JPEG without converting to RGB. All of the libraries used are listed at the bottom.

I know, but even VS's support in this respect only applies to jpeg, over the libjpeg(-turbo) library. Which falls under my format-specific point. I'm proposing a new plugin (e.g. JPEGSource) for this purpose.

[Plorkyeran]

Does it really return the original image data unchanged? Or maybe it gets RGB data internally from the format's lib, then ffmpeg converts back to planar on its own?
libavcodec doesn't even have a way to convert the data to RGB, and I have no idea why you'd expect it to be automatically converting to RGB and back.

[jpsdr]

You could just sign extend your edx into rcx to solve this "problem". For example (untested):

I've already used sometimes movzd in my VDub filters, but haven't searched yet if something else existed.
You spare me time searching, and this will probably work.
Thanks.

[SEt]

I'm proposing a new plugin (e.g. JPEGSource) for this purpose.
Actually I have working very nice JPEG decoder that can be ported to Avisynth. It not just keeps original chroma subsampling, but also supports "reconstruction" that accurately suppresses typical JPEG artifacts and can be done only as part of the decoding stage. Won't be open-source though.

It's currently implemented as plugin (http://plugring.farmanager.com/plugin.php?pid=693) for Far Manager (http://www.farmanager.com/download.php?l=en). It has some documentation, but only in Russian.

[jpsdr]

I think i may have discoverd an issue with x64 version, when trying to port the nnedi3 to x64.
When investigating, i've disable the use of ASM function, and fixed number of cpu (or threads) to 1.

I've the following result, code is around lines 1168.

With this :

else if (vi.IsYV12())
{
for (int i=0; i<ct; ++i)
{
v = new nnedi3(v.AsClip(),i==0?1:0,true,true,true,true,nsize,nns,qual,etype,pscrn,threads,opt,fapprox,env);
env->ThrowError("Ok");
v = env->Invoke("TurnRight",v).AsClip();
// always use field=1 to keep chroma/luma horizontal alignment
v = new nnedi3(v.AsClip(),1,true,true,true,true,nsize,nns,qual,etype,pscrn,threads,opt,fapprox,env);
v = env->Invoke("TurnLeft",v).AsClip();
}

the script stoped with "Ok" error.

With this :

else if (vi.IsYV12())
{
for (int i=0; i<ct; ++i)
{
v = new nnedi3(v.AsClip(),i==0?1:0,true,true,true,true,nsize,nns,qual,etype,pscrn,threads,opt,fapprox,env);
v = env->Invoke("TurnRight",v).AsClip();
env->ThrowError("Ok");
// always use field=1 to keep chroma/luma horizontal alignment
v = new nnedi3(v.AsClip(),1,true,true,true,true,nsize,nns,qual,etype,pscrn,threads,opt,fapprox,env);
v = env->Invoke("TurnLeft",v).AsClip();
}

script stop with "acces violation" error.

With this :

else if (vi.IsYV12())
{
for (int i=0; i<ct; ++i)
{
v = new nnedi3(v.AsClip(),i==0?1:0,true,true,true,true,nsize,nns,qual,etype,pscrn,threads,opt,fapprox,env);
//v = env->Invoke("TurnRight",v).AsClip();
// always use field=1 to keep chroma/luma horizontal alignment
v = new nnedi3(v.AsClip(),1,true,true,true,true,nsize,nns,qual,etype,pscrn,threads,opt,fapprox,env);
env->ThrowError("Ok");
v = env->Invoke("TurnLeft",v).AsClip();
}

script stoped with "Ok" error.

With this :

else if (vi.IsYV12())
{
for (int i=0; i<ct; ++i)
{
v = new nnedi3(v.AsClip(),i==0?1:0,true,true,true,true,nsize,nns,qual,etype,pscrn,threads,opt,fapprox,env);
//v = env->Invoke("TurnRight",v).AsClip();
// always use field=1 to keep chroma/luma horizontal alignment
v = new nnedi3(v.AsClip(),1,true,true,true,true,nsize,nns,qual,etype,pscrn,threads,opt,fapprox,env);
v = env->Invoke("TurnLeft",v).AsClip();
env->ThrowError("Ok");
}

script stopped with "acces violation" error.

and with this :

else if (vi.IsYV12())
{
for (int i=0; i<ct; ++i)
{
v = new nnedi3(v.AsClip(),i==0?1:0,true,true,true,true,nsize,nns,qual,etype,pscrn,threads,opt,fapprox,env);
//v = env->Invoke("TurnRight",v).AsClip();
// always use field=1 to keep chroma/luma horizontal alignment
v = new nnedi3(v.AsClip(),1,true,true,true,true,nsize,nns,qual,etype,pscrn,threads,opt,fapprox,env);
//v = env->Invoke("TurnLeft",v).AsClip();
}
env->ThrowError("Ok");

script stopped with "Ok" error.

So, according the results, i have the feeling it's something related to the x64 version of avisynth, as crash seems to occur during call of internal avisynth functions.

[real.finder]

You are not the first to suggest that some data types should be changed, but we cannot do so without breaking existing x64 filters.



um, existing x64 filters most of them (maybe all) in 2.5

since there is no avs 2.6 x64 around, and 2.6 x64 filters is just a few, actually the existing is:- mask tools 2 and TP7 filters (http://forum.doom9.org/showthread.php?t=169832) and last is LaTo filters, it can be update

so, 2.6 in x64 can be change alone? or it can't?

[steptoe]

I'm trying to work out how to use the updated RGTools.dll in place of the RemoveGrain package, but unsure if its just a case of replacing all calls in the removegrain package with rgtools.dll or if the syntax is different

At its most basic, calling for example RemoveGrain(17) or RemoveGrain(2) would this simply now be RGTools(17) or RGTools(2), and loading RGTools.dll instead of RemoveGrain.dll via the 'loadplugin' command

The docs for the updated plugins for avisynth+ are a bit sparse, and don't really suggest if they are direct drop in replacement or if a bit more work may be needed if the original plugins are called within scripts


Scripts that call various parts of the removegrain package, such as RemoveDirtMC, RemoveNoiseMC or RemoveNoiseMC_SE scripts are giving me a headache trying to update them to use RGTool. These all call different parts of the RemoveGrain package, such as the clense function for temporal noise/grain removal

Thanks

[TurboPascal7]

First of all, you're posting in a wrong thread. This (http://forum.doom9.org/showthread.php?t=169832) is the right one.
First 24 modes of RGTools should be drop-in replacement of the old removegrain plugin. Same goes for repair (24 modes), verticalcleaner (3 modes) an clense (normal, backward, forward). RGTools doesn't have some of the parameters in Clense so you'll have to drop those if you want to use it.

[Zathor]

Besides proudly wielding the version number "0.1 (r1555)"[...]
Thanks for the new version numbers. I am not sure if it is intended, but the information in the dll itself and VersionString() in AviSynth do not match. Currently there is:
VersionString(): AviSynth+ 0.1 (r1561, x86)
FileProperties.ProductName: AviSynth+ 2.6
FileProperties.FileVersion: 2, 6, 0, 5
(and also some others - maybe you also want to change the home page)

[ultim]

I think i may have discoverd an issue with x64 version, when trying to port the nnedi3 to x64.
...

No, it is actually a bug in nnedi3's sources, it is using an avisynth.h that is not x64 compatible. The bug is in AVSValue's Assign() function in nnedi3, you should take the working implementation from Avs+ sources to fix it.

p.s. This is why I am against every plugin having a separate copy of the avisynth header. With possibly all other libraries around the world, if someone wants to compile an extension or application for that lib, they just use the headers from the library's sources by adding the core's public headers path to the compiler include dirs. Why can't AVS plugin devs do that too?

[Gavino]

This is why I am against every plugin having a separate copy of the avisynth header. With possibly all other libraries around the world, if someone wants to compile an extension or application for that lib, they just use the headers from the library's sources by adding the core's public headers path to the compiler include dirs. Why can't AVS plugin devs do that too?
That's my view too, but TurboPascal7 (here) disagrees.

[ultim]

That's my view too, but TurboPascal7 (here) disagrees.

About 2-3 weeks ago I helped him debug one of his sources and that (too) turned out to be caused by a wrong avisynth.h header in the plugin. So he should have first-hand experience about the issues his method is causing. But here is my reflection on his arguments:

1) Avisynth folder might be different on different systems, someone who uses your project file will need to modify that. One might not even have the required version installed. Of course if you're writing closed source plugins or hope to avoid contributors, this is "fine".

Yes, VC++ project files will need to be modified if a stranger tries to compile your plugin. But the compiler will tell you instantly and unambiguously that avisynth.h is not found, and the dev will know right away that the include path needs to be modified. But my argument is not that this is an easy fix. The real argument here is that fixing up the include path is much easier than trying to debug runtime problems caused by mismatched headers. Such runtime problems, as both recent and past experiences show, do happen a lot.

2) It shows you the exact version of avisynth.h used to build a plugin. Just earlier this week I had to fix a plugin that didn't work in x64 mode because it still used AvisynthPluginInit2 even though it came with 2.6 header. This header being included helped me to identify the problem a lot faster than I could do with other methods.
Most times when you are compiling the plugin, you are not interested in which header version was used to compile it earlier. You will most probably have your own idea what AVS version you are compiling for, and that's all. Besides, the scenario described above is kind of like the self-fullfilling prophecy: Had the plugin been forced to correctly update to the latest header, the above mentioned bug wouldn't have happened at all. So frankly, I cannot accept this argument.

Which is kind of the point of always using the header from the avs(+) sources instead of making copies of it: 1) During compilation it will be obvious which Avs the plugin will be compatible with: that whose sources were used. 2) It'd avoid out-of-date headers of old plugins. 3) Plugins would automatically get (after a recompilation) all the fixes that go into the interface upstream. 4) It'd make it a lot less easy to mismatch header version with the intended/commonly used Avs version.

[jpsdr]

No, it is actually a bug in nnedi3's sources, it is using an avisynth.h that is not x64 compatible. The bug is in AVSValue's Assign() function in nnedi3, you should take the working implementation from Avs+ sources to fix it.

Thanks.

Any new release (ei 1576...) scheduled ?

[LoRd_MuldeR]

Yes, VC++ project files will need to be modified if a stranger tries to compile your plugin. But the compiler will tell you instantly and unambiguously that avisynth.h is not found, and the dev will know right away that the include path needs to be modified.

Why not simply use an include path like "$(AVISYNTH_ROOT)\include" in your solution? Then the developer just needs to setup the environment variable AVISYNTH_ROOT properly and the solution file does not need to be modified at all...

[ultim]

Any new release (ei 1576...) scheduled ?

Actually, I was just about to :)

So enjoy r1576. This is the fourth bugfix release in the current stable series, bringing you:

Reduced memory consumption in resizers.
The fix for this (http://forum.doom9.org/showthread.php?p=1657586#post1657586) crash.
A fix for correct detection of AVX capability.
A small "for"-loop change for compatibility with Gavino's original version.
A fix for bad alignment in the crop filter.
A compatiblity fix in BitBlt.
And a fix for a bug that resulted in autoloaded functions sometimes taking precedence over a non-autoloaded version.


Okay, so much for the bugfixes. Boring but at least usefull. If you want something cool though, try out and help me test the new caching system in this experimental build (http://goo.gl/e0VFYn). Make sure you rename the file to "avisynth.dll" if you try it out.

The experimental build is mostly the same as the just released r1576, except that it has the new caches, so if you're doing comparisons, please compare the experimental build to the r1576 release in this post. The new caches have been written from scratch with MT in mind, and although MT is not yet active in this build, the new caches will (or should) provide similar performance to the stable release, but with significantly reduced memory consumption. Let me know your experiences. The sooner I can deem the new caches "good enough", the sooner we'll see MT ;)

[jpsdr]

Bad news... The x64 version of r1576 is broken, i've rolled-back to r1561.
(Good new, is that i've now been able to make my nnedi port working on x64...).

[Groucho2004]

The x64 version of r1576 is broken
That's an amazingly detailed bug report.

Anyway, the 64 Bit dll is a debug build and the correct runtime (msvcp110d.dll) is missing.

[jpsdr]

Any avisynth script will generate an "unknow" error... Anyone who install and test it will see.

[ultim]

Gomen. I updated the binaries with release builds of the x64 version.

[DrZine]

Today I decided to try out avs+ for the first time. Everything seems to be stable so I did some benchmark comparisons. I'm on an AMD system and since hardly any testing is being done with AMD I thought I might share. I also tested out the new caching build just posted a few days ago while I was at it.

My system is a FX8350 @4.4ghz with 8gb of ram @1600mhz running Win7 Pro x64. All builds of avs+ are x86. I ran 2 different scripts with AVSmeter 1.7.4.

The first script I ran was a test script Groucho2004 posted up a few pages back. Going from Alpha5 to avs+ r1576 was a 50% speed increase! I'm guessing this has more to do with the improved core filters than anything else. From r1576 to the new cache build r1595 was a bit of a head scratcher though. The overall speed dropped just a hair while a second thread was created. Is this supposed to happen? As far as the memory usage it dropped by 155MB. Very nice.

[Script]
colorbars(width = 1920, height = 1080, pixel_type = "yv12").killaudio().assumefps(24000, 1001)

trim(0,499)
fadeio(248)
trim(0,499)

spline36resize(width() - 64, height() - 64).turnleft()
spline64resize(width() + 64, height() + 64).turnright()
bicubicresize(width() - 64, height() - 64).fliphorizontal()
sincresize(width() + 64, height() + 64).flipvertical()

a = tweak(hue=33)
u_chroma = blankclip(utoy(a), color=$808080)
ytouv(u_chroma, a.vtoy)
mergeluma(a)
tweak(hue=-33)

temporalsoften(4,4,8,15,2)
limiter(16, 235, 16, 240)
levels(0, 1, 255, 16, 235)
scriptclip("subtitle(string(ydifferencefromprevious))")
a=selectevery(3, 0).addborders(0,0, 16,0).crop(0,0, -16,0)
b=selectevery(3, 1).addborders(0,0, 32,0).crop(0,0, -32,0)
c=selectevery(3, 2).addborders(0,0, 64,0).crop(0,0, -64,0)
interleave(a, b, c)

----------------------------------------------------

[General info]
Log file created with: AVSMeter 1.7.4 (AVS 2.6, x86)
Avisynth version: AviSynth 2.60, build:Sep 18 2013 [17:36:36]


[Clip info]
Number of frames: 500
Length (hhh:mm:ss.ms): 000:00:20.854
Frame width: 1920
Frame height: 1080
Framerate: 23.976 (24000/1001)
Interlaced: No
Colorspace: YV12


[Runtime info]
Frames processed: 500 (0 - 499)
FPS (min | max | average): 2.16 | 11.68 | 4.77
CPU usage (average): 12%
Thread count: 1
Physical Memory usage (peak): 467 MB
Virtual Memory usage (peak): 464 MB
Time (elapsed): 000:01:44.901

----------------------------------------------------

[General info]
Log file created with: AVSMeter 1.7.4 (AVS 2.6, x86)
Avisynth version: AviSynth+ 0.1 (r1576, x86)


[Clip info]
Number of frames: 500
Length (hhh:mm:ss.ms): 000:00:20.854
Frame width: 1920
Frame height: 1080
Framerate: 23.976 (24000/1001)
Interlaced: No
Colorspace: YV12


[Runtime info]
Frames processed: 500 (0 - 499)
FPS (min | max | average): 3.03 | 13.95 | 7.55
CPU usage (average): 12%
Thread count: 1
Physical Memory usage (peak): 485 MB
Virtual Memory usage (peak): 480 MB
Time (elapsed): 000:01:06.221

----------------------------------------------------

[General info]
Log file created with: AVSMeter 1.7.4 (AVS 2.6, x86)
Avisynth version: AviSynth+ 0.1 (r1595, x86)


[Clip info]
Number of frames: 500
Length (hhh:mm:ss.ms): 000:00:20.854
Frame width: 1920
Frame height: 1080
Framerate: 23.976 (24000/1001)
Interlaced: No
Colorspace: YV12


[Runtime info]
Frames processed: 500 (0 - 499)
FPS (min | max | average): 1.49 | 13.71 | 7.44
CPU usage (average): 12%
Thread count: 2
Physical Memory usage (peak): 330 MB
Virtual Memory usage (peak): 324 MB
Time (elapsed): 000:01:07.238

The second script I did was a bit of random calls to the Dither Package 1.24.0 on a 1080p blu ray rip I've been playing around with. I was just messing around with some resizing and a few basic filter calls on a 16bit stacked clip. Speed stayed basicly the same between all 3 builds in the 21fps range. The big story here is memory usage. 592MB for Alpha5, 714MB for r1576, and 146MB for r1595! :eek: I had to rerun that last one with my system monitor. 146MB is a massive improvement! It was also the fastest test run.

[Script]
LoadPlugin("D:\Installs\Media\Video Encoding\MeGUI_2050_x86\tools\ffms\ffms2.dll")
FFVideoSource("D:\Encodes\Kara no Kyoukai\5\Kara no Kyoukai 5.mkv", threads=1)
Trim(2788, 3938)
Dither_convert_8_to_16()
Dither_resize16(848, 480)
Dither_removegrain16()
Dither_SmoothGrad()
Dither_resize16(1920, 1080)
DitherPost()

----------------------------------------------------

[General info]
Log file created with: AVSMeter 1.7.4 (AVS 2.6, x86)
Avisynth version: AviSynth 2.60, build:Sep 18 2013 [17:36:36]


[Clip info]
Number of frames: 1151
Length (hhh:mm:ss.ms): 000:00:48.006
Frame width: 1920
Frame height: 1080
Framerate: 23.976 (24000/1001)
Interlaced: No
Colorspace: YV12


[Runtime info]
Frames processed: 1151 (0 - 1150)
FPS (min | max | average): 13.16 | 22.03 | 21.47
CPU usage (average): 27%
Thread count: 13
Physical Memory usage (peak): 592 MB
Virtual Memory usage (peak): 595 MB
Time (elapsed): 000:00:53.622

----------------------------------------------------

[General info]
Log file created with: AVSMeter 1.7.4 (AVS 2.6, x86)
Avisynth version: AviSynth+ 0.1 (r1576, x86)


[Clip info]
Number of frames: 1151
Length (hhh:mm:ss.ms): 000:00:48.006
Frame width: 1920
Frame height: 1080
Framerate: 23.976 (24000/1001)
Interlaced: No
Colorspace: YV12


[Runtime info]
Frames processed: 1151 (0 - 1150)
FPS (min | max | average): 2.86 | 23.25 | 21.33
CPU usage (average): 25%
Thread count: 13
Physical Memory usage (peak): 714 MB
Virtual Memory usage (peak): 716 MB
Time (elapsed): 000:00:53.966

----------------------------------------------------

[General info]
Log file created with: AVSMeter 1.7.4 (AVS 2.6, x86)
Avisynth version: AviSynth+ 0.1 (r1595, x86)


[Clip info]
Number of frames: 1151
Length (hhh:mm:ss.ms): 000:00:48.006
Frame width: 1920
Frame height: 1080
Framerate: 23.976 (24000/1001)
Interlaced: No
Colorspace: YV12


[Runtime info]
Frames processed: 1151 (0 - 1150)
FPS (min | max | average): 12.87 | 22.27 | 21.71
CPU usage (average): 26%
Thread count: 13
Physical Memory usage (peak): 146 MB
Virtual Memory usage (peak): 148 MB
Time (elapsed): 000:00:53.016

Besides the tests I ran. All I got to say is great job team with avs+! Barring any stability issues I might run into, I'm going to just keep using the experimental build from now on untill the next build comes along.

[Groucho2004]

My system is a FX8350 @4.4ghz with 8gb of ram @1600mhz running Win7 Pro x64. All builds of avs+ are x86. I ran 2 different scripts with AVSmeter 1.7.4.

The first script I ran was a test script Groucho2004 posted up a few pages back. Going from Alpha5 to avs+ r1576 was a 50% speed increase! I'm guessing this has more to do with the improved core filters than anything else. From r1576 to the new cache build r1595 was a bit of a head scratcher though. The overall speed dropped just a hair while a second thread was created. Is this supposed to happen? As far as the memory usage it dropped by 155MB. Very nice.
Comparing your numbers for the first script with my i5 2500K @ 4GHz:
Avisynth version: AviSynth 2.60, build:Sep 18 2013 [17:36:36]
FPS (min | max | average): 3.72 | 22.39 | 10.54 (+121%)

Avisynth version: AviSynth+ 0.1 (r1576, x86)
FPS (min | max | average): 4.53 | 26.45 | 13.06 (+73%)

Avisynth version: AviSynth+ 0.1 (r1595, x86)
FPS (min | max | average): 2.27 | 26.31 | 13.11 (+76%)

Memory usage was about the same. I did not expect Intel to be so much faster. However, this is just one script scenario and others may differ. Also, AVS+ optimizations seem to be a bit more AMD friendly. :)
One more note - The experimental DLL created 5 threads (well, 4 threads really, 1 is for the calling process, AVSMeter) in my case, not 2. Odd.

[innocenat]

Just a note , but none of the dev have AMD machine, thus most code are optimized for intel processors. Also, Sandy Bridge and later are insanely powerful.

[ryrynz]

Yeah, Intel completely dominates in this area, i7's with hyperthreading enabled using an MT build are most certainly king, I wouldn't buy an AMD machine for Avisynth processing, I highly doubt AMD's 8 cores
could even compete with most i5's let alone i7's and they'd use more energy to accomplish the same task too. That said, I wouldn't mind seeing a thread that compared various scripts on different processors so that would could make
more informed decisions on what hardware to buy.

[ultim]

DrZine, Groucho2004: Yes, those threads are normal. They are threadpool threads from the MT version, but in this particular build they just sleep/idle and do nothing, because this build is only meant to test the caches.

[BBA163]

When you can fix the avs+ load bug in megui or avspmod?
avspmod I try to change the licence ,but it's not work:scared:

[Zathor]

When you can fix the avs+ load bug in megui or avspmod?
avspmod I try to change the licence ,but it's not work:scared:
Can you please describe this avs+ load bug? At least I do not know it.

[Evil_Burrito]

Perhaps BBA163 is referring to, "An official release which will add support for 64-bit AviSynth+ is expected soon." As mentioned on the avisynth+ get started page.

Btw r1595 x86 has generally been working fine for me with avspmod and encoding. Mostly just a few cache_child errors from incompatible filters.

[tObber166]

Everytime I start up AvsPmod I get this error:

"Error parsing plugin string at position 0:"

It doesn't crash or anything, and all my plugins and filter work just fine.

AviSynthPlus-r1576

[DrZine]

The fix for that is here

http://forum.doom9.org/showthread.php?p=1648046#post1648046

[tObber166]

Thanks! ;)

[BBA163]

load avs in megui it's was crash:scared:
(My English is bad and reply was slow,sorry)

[innocenat]

load avs in megui it's was crash:scared:
(My English is bad and reply was slow,sorry)

Please provide more information: script used, etc.

[BBA163]

Just use ffvideosource,can you load avisynth+'s avs file in megui?
If you can…that's my problem

[Zathor]

Yes, I can load such avs files without any problems.

[ajp_anton]

What happened to index.htm in the documentation folder?

[ultim]

What happened to index.htm in the documentation folder?

MIA (by mistake).

ps.: Wow, there *are* some people reading the offline docs. Well good for you, it's about to get updated ;)

[BBA163]

So strange with my PC to load it…

[ultim]

So here is a small update on the status of the MT version of avs+. I'm planning on stabilizing this ASAP now, merging it to my usual main branch this week. This also means that all pull requests and even non-MT work will end up in an MT-enabled build from now on.

I spent most of my development time last weekend hunting down the source of some uninitialized memory, so I got somewhat delayed, but hopefully by the end of this week I'll publish a new experimental build, and for plugin developers most importantly, I'm also pushing the whole work to GitHub. There are only a couple of things left on my to-do list, with one exception (addressing a theoretical deadlock) just build- or performance enhancements.

A slight difference to previous plans is that I'm splitting the current threadpool into two, one for the core threads and one for filter-initiated threads. This is made necessary to avoid the previously mentioned theoretical deadlock, 'coz any other alternative (instead of the threadpool split) that comes to my mind would involve some sophisticated (and error-prone!) locking trickery with questionable benefits. The split will simplify code a lot compared to other fixes, and should not hurt performance.

It would be nice to have the new documentation ready by then, extended with all the Avs+ and new MT-specific stuff, so that interested developers could consult it at once, but that won't be done by this week for sure. That's because even though qyot has done most of the hard work on converting the docs format (oh Lord it's awesome), there is still some left to be done, and coupled with my other to-do items, the weekend just won't be enough.

One way you guys can help me to make MT usable as soon as possible now, is to help collect what MT modes existing filters need. The three supported MT modes are the same as for SEt's version, 1 - no protection needed at all, 2 - multiple filter instances per call (one per thread), and 3 - just one filter for all threads but protected by exclusive area. It'd be our collective interest to collect what each filter needs, or if there is maybe already a list like that, to extend/update it. The rules could ship with avs+ then to make it readily usable. So can I leave you to assemble this list? Maybe on a Wiki page?

Active plugin developers can include the rules for their filters inside their own plugin, so those do not need to be on the list. They will be able to start including the functionality as soon as I push the new avisynth.h this week. Moreover, they can do so while still staying compatible with classic (non-plus) Avs.

[innocenat]

Does this mean new plugin won't work with 2.6a5 anymore?

[ultim]

Does this mean new plugin won't work with 2.6a5 anymore?

It will be compatible. New plugins will still work with 2.6a5. The only limitation is that they will need a 2.6 header, so it won't work with 2.5.

[real.finder]

for me mode 2 work in 90% of filters, Especially in the last SEt builders, by now I use mode 2 only even with tdecimate 2pass vfr Which were not compatible in the past

Waiting for the new header and MT

thanks :)

[MasterNobody]

As I understand due to new way of auto loading plugins (pending them to the later time) avs_function_exists from C-interface fails to find out that DirectShowSource is available. This result in fail to open video files in x264 when you specify as input media file (any mkv file and --demuxer avs to be sure not to use lavf input) and not the avs script file. Is there is way to fix it from AviSynth+ side or what have to be added from x264 side so avs_function_exists will work?

[qyot27]

As I understand due to new way of auto loading plugins (pending them to the later time) avs_function_exists from C-interface fails to find out that DirectShowSource is available. This result in fail to open video files in x264 when you specify as input media file (any mkv file and --demuxer avs to be sure not to use lavf input) and not the avs script file. Is there is way to fix it from AviSynth+ side or what have to be added from x264 side so avs_function_exists will work?
It's actually failing to find any source filters, not just DirectShowSource.
>x264 --preset ultrafast --crf 18 -o test.mkv "Qyot27 - Daybreak [XviD+MP3].mkv" --frames 10 --demuxer avs --verbose
avs [debug]: using avisynth version 2.60
avs [info]: trying FFmpegSource2... not found
avs [info]: trying DSS2... not found
avs [info]: trying DirectShowSource... not found
avs [error]: unable to find source filter to open `Qyot27 - Daybreak [XviD+MP3].mkv'
x264 [error]: could not open input file `Qyot27 - Daybreak [XviD+MP3].mkv'

So I doubt it's related to the C-plugin autoloading (the copy of FFMS2 that failed is a C-plugin). Looking at avs_core/core/PluginManager.cpp, it seems to be under avsplus' new experimental IScriptEnvironment2 class. I think that's more likely to be the issue, but I don't know anything about C++.

[MasterNobody]

Looks like I found the way how to fix it from x264 side by adding invoke of AutoloadPlugins(): patch (http://privatepaste.com/faf965559d)

[ultim]

Avs+ autoloads plugins if any of the following happens:
- AutoloadPlugins() is called
- LoadPlugin() is called
- A yet unknown (non-internal) function is called

avs_function_exists does not find the external source filter in this case because none of the above happened. So MasterNobody's patch is the right thing to do.

[Gavino]

Avs+ autoloads plugins if any of the following happens:
- AutoloadPlugins() is called
- LoadPlugin() is called
- A yet unknown (non-internal) function is called

avs_function_exists does not find the external source filter in this case because none of the above happened. So MasterNobody's patch is the right thing to do.
How about having env->FunctionExists() (which is called by avs_function_exists) autoload plugins if a yet unknown function name is passed? Then it is transparent to client applications.

[ultim]

How about having env->FunctionExists() (which is called by avs_function_exists) autoload plugins if a yet unknown function name is passed? Then it is transparent to client applications.

Hmm, I didn't like the idea at first, but thinking about it some more, it does make sense. The fact that FunctionExists() never autoloads while a function call can do it means that it is possible to call functions for which FunctionExists() returned false. That is not only inconsistent, but it can be argued it is a bug.

On the other hand, the reason I'm not perfectly satisfied with FunctionExists() autoloading is because FunctionExists() has the semantics of a getter function. It is pretty unexpected that it changes the internal state greatly (even if we are talking about non-observable state). I have to conclude though that this rather ideological argument is outweighed by the proposed solution's practical relevance.

So I guess I will correct this in Avisynth+ after all.

[real.finder]

hi

I have two aWarpSharp in autoload folder, aWarpSharp and aWarpSharp2 by SEt

In normal avs if I use


LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\aWarpSharp.dll")
someSource("E:\New.mkv")
aWarpSharp


green border will appear which mean that old aWarpSharp is using

but in avs+ only aWarpSharp2 will be used

I use this method in other cases also

[ultim]

hi

I have two aWarpSharp in autoload folder, aWarpSharp and aWarpSharp2 by SEt

In normal avs if I use


LoadPlugin("C:\Program Files\AviSynth 2.5\plugins\aWarpSharp.dll")
someSource("E:\New.mkv")
aWarpSharp


green border will appear which mean that old aWarpSharp is using

but in avs+ only aWarpSharp2 will be used

I use this method in other cases also

Are you using the latest release (r1576)? This is something that should have been fixed there.

[real.finder]

Are you using the latest release (r1576)? This is something that should have been fixed there.

yes, and even r1595 the same thing

[ultim]

yes, and even r1595 the same thing
the only difference between 1576 and 1595 is the cache, so if the former one didnt work, the latter one is no surprise.

anyway, thx for reporting, i'll investigate.

[list]

In Avisynth 2.58, i used to make it portable, and just place avisynth.dll and devil.dll beside my mplayerc player and play any avs's, just needed to run this .reg with these values for "installing" on my system manually, without any installer.
The reg file contains these register key values:
REGEDIT4

[HKEY_CLASSES_ROOT\avifile\Extensions\AVS]
@="{E6D6B700-124D-11D4-86F3-DB80AFD98778}"

[HKEY_CLASSES_ROOT\Media Type\Extensions\.avs]
"Source Filter"="{D3588AB0-0781-11CE-B03A-0020AF0BA770}"

[HKEY_CLASSES_ROOT\CLSID\{E6D6B700-124D-11D4-86F3-DB80AFD98778}]
@="Avisynth"

[HKEY_CLASSES_ROOT\CLSID\{E6D6B700-124D-11D4-86F3-DB80AFD98778}\InProcServer32]
@="avisynth.dll"
"ThreadingModel"="Apartment"

[HKEY_CLASSES_ROOT\Wow6432Node\Media Type\Extensions\.avs]
@=""
"Source Filter"="{D3588AB0-0781-11CE-B03A-0020AF0BA770}"

[HKEY_LOCAL_MACHINE\SOFTWARE\CLASSES\WOW6432NODE\CLSID\{E6D6B700-124D-11D4-86F3-DB80AFD98778}\INPROCSERVER32]
@="AviSynth.dll"
"ThreadingModel"="Apartment"

But now, this dosn't work with avsPlus. So i needed to run the full install to make it work.
What would be the new .reg equivalent now to meke it portable if possible?
BTW the project looks promising :D

[qyot27]

I'm guessing that due to the use of C++11 (the future header and whatnot), the MT additions will bump the compiler requirements up to VS 2012 or higher?

[innocenat]

I'm guessing that due to the use of C++11 (the future header and whatnot), the MT additions will bump the compiler requirements up to VS 2012 or higher?

Yes, the MT additions will bump the compiler requirement to VS2012. (It's actually already available in MT branch on GitHub)

[qyot27]

Yes, the MT additions will bump the compiler requirement to VS2012. (It's actually already available in MT branch on GitHub)
Okay. That's why I asked; it failed to build under 2010.

[jpsdr]

Will avs+ still working under Windows XP after MT...?

[TurboPascal7]

Will avs+ still working under Windows XP after MT...?

Yes. Public builds have been compiled with VS2012 since the beginning I think.

But I'm afraid it won't work in wine anymore, at least my threaded plugins don't.

[ultim]

OK, seeing that words about the MT version is beginning to spread on the forums, I'd like to point out that it is very experimental and still has some problems. I am very gratefull if anybody wants to test and give feedback / problem reports, but if your motivation is not to test but to start using it earlier, don't. The quality is not there yet. I'll announce in this thread when it reaches a quality niveau worthy of public testing. That being said, you can find the experimental builds on http://goo.gl/e0VFYn . Just remember, don't use it until you're prepared that it will break.

[easyfab]

Do we need some setmtmode() or mt() or is it automaticaly multithreaded ?

[ultim]

Do we need some setmtmode() or mt() or is it automaticaly multithreaded ?

You need to enable MT operation in the script. But let's just first test the single-threaded mode...

[naoan]

You need to enable MT operation in the script. But let's just first test the single-threaded mode...

Is SetFilterMTMode disabled on that build?

[innocenat]

Is SetFilterMTMode disabled on that build?

There is no public MT build yet.

[naoan]

Oh sorry, I thought it is with experimental.

[TurboPascal7]

Okay, it appears I was right and new experimental threaded builds do not work in wine. If anyone figures how to run it - please feel free to post the instructions. Otherwise poor linux users won't be able to enjoy any newer build of avs+ until native support is implemented and plugins are ported or this gets fixed in wine.

[ultim]

Okay, it appears I was right and new experimental threaded builds do not work in wine. If anyone figures how to run it - please feel free to post the instructions. Otherwise poor linux users won't be able to enjoy any newer build of avs+ until native support is implemented and plugins are ported or this gets fixed in wine.

It is most probably because of the threading bits of the VC++2012 runtime.

Possible solutions:
- Get the VC++2012 runtime somehow correctly working on Wine.
- Submit bug report to Wine and have it fixed.
- Wait for native Linux support in avs+ so that Wine won't be needed.

[dvdan23]

The bug which prevents vc++2012 from installing under WINE was reported in 2009

http://bugs.winehq.org/show_bug.cgi?id=17273

Waiting for that to get fixed seems like a long shot, though it has been targeted for 1.8 for whatever that's worth.

As awesome as it sounds, waiting for "native Linux support" seems fraught with peril also, since many have attempted this in the past and not quite made it (Avisynth3, avxsynth, vapoursynth).

Ultim, have you taken some time to assess the downsides of limiting yourself to VC++2010 SP1? if you would consider it at least one linux user will rejoice.

http://appdb.winehq.org/objectManager.php?sClass=application&iId=5766

[TurboPascal7]

Actually, getting native linux support on the core side is not hard. You can expect the core to work in its full power, unlike avxsynth that removed optimizations and some filters which didn't have C versions, avisynth 3 that didn't happen or vsynth, which is vsynth. You'll also get some essential plugins from avs+ modernization efforts thread and maybe some more. But of course it'll be quite a while before any serious scripts like qtgmc can run on linux. As usual - patches welcome. :)

To replace vs2012 features with vs2010, one will need to introduce some huge dependency like boost. This was discussed already and we decided that using C++11 is a better option. Also Myrsloik promised to kill me if we don't go C++11 and you don't want that, do you? >__>

[ultim]

It is pretty much as Turbo explained. I have originally implemented avs+MT with only boost, and thus I could compile the code with VC++2010. Because of the threading part I was relying on the non-header-only libraries of boost (but even with headers only, boost is a beast considering its size). So later the requirement was bumped to VC++2012 in order to remove the boost dependency, because C++11 implements everything I used from boost, but VC++2010 doesn't yet implement the necessary parts of the standard.

Anyway, I didn't foresee that it would break Wine-support, that hit me unexpected. IMHO we should try to put some pressure on the Wine folks, and depending on what their reaction is, we either win, or I can step back and pull in boost again. I'd rather not though unless it is really necessary, so let's try convincing the Wine folks first. It's on their to-do list anyway :)

[dvdan23]

Turbopascal7, I certainly wouldn't want Myrsloik to kill you. I also don't want to see Linux compatibility killed though.

As I missed this discussion, I take it "requiring a huge dependency" was deemed the greater evil vs. breaking under Linux? FWIW, that wouldn't have been my vote.

If boost libraries are indeed of comparable quality, it seems to me it's a dependency either way. In my case, the dependency of windows is a far greater evil than the dependency on a free, portable library like boost. the boost licensing is pretty unrestrictive. How's the performance?

If the death of Avisynth+ on Linux is truly unavoidable, and last minute calls to the governor can't do anything to save it, might it be feasible to keep a single-threaded branch alive on the git hub and/or make a periodic build for folks who can't run VC2012++? if other solutions pan out that'll be great but I suspect that might be a few years.

[TurboPascal7]

If boost libraries are indeed of comparable quality, it seems to me it's a dependency either way. In my case, the dependency of windows is a far greater evil than the dependency on a free, portable library like boost. the boost licensing is pretty unrestrictive. How's the performance?
Dependency on windows has always been there, nothing really changed on that front. The fact that wine has bugs is, well, a wine's problem. I don't think it's appropriate to make life of 99% of avisynth users and all contributors harder to compensate for it. Dependencies on external libraries over standard are pretty much always worse - it's harder to setup build enviroment, it takes longer to compile and usually produces large binaries.

I understand that it's quite unfortunate that avs+ won't work on linux, but it can't be helped. You could always setup a VM too. And if the demand for linux support is high, of course there will be some developers willing to port plugins to this platform.

If the death of Avisynth+ on Linux is truly unavoidable, and last minute calls to the governor can't do anything to save it, might it be feasible to keep a single-threaded branch alive on the git hub and/or make a periodic build for folks who can't run VC2012++? if other solutions pan out that'll be great but I suspect that might be a few years.
Not to mention that it might be quite hard to merge the changes, there's one more problem with this solution - the plugins. With MT implemented in avs+ core, internal threading in plugins is strongly discouraged and you can expect it to get removed from most "updated" plugins, leading a lot worse performance with singlethreaded versions of avisynth. Asking developers to maintain branches with internally threaded code in all of these plugins is way too much. So you'll be either enjoying crippled performance or will have to avoid newer versions of such plugins.

[dvdan23]

Learned something new from the Wine folks. Apparently the approach they are taking is to build a complete VC++ library replacement so you don't need to install VC++ runtimes. Apparently if the "Microsoft Foundation Classes" can be avoided, there's a better shot at Avisynth++ working under WINE.

From the bug $17273 "Applications affected" page;

NOTE: There is an ongoing effort in Wine to provide a complete Microsoft Visual C++runtime library replacement - removing the need of installing the Microsoft VC++ redistributable packages. This is a work in progress hence you might still need to install Microsoft VC++ runtime.

If the application uses MFC library (Microsoft Foundation Classes), there will be no replacement hence installation of Microsoft Redistributable is a requirement for applications and games to work!

TurboPascal7, I'm not going to argue with you because I'm thrilled with the work you are doing making more plugins 64-bit. I'll tell you though, I'm much more interested in stable, working, 64-bit non-threaded avisynth plugins than threaded ones. Keep up the good work.

[TurboPascal7]

The only thing I wanted to say is that making the wine guys fix their stuff is a lot better than limiting program functionality in any way. I'm sure if users complain enough, they'll get it working sooner than later. ;)

[qyot27]

It would appear that a deeper fix would be needed anyway, since while it is possible to yank the .dlls themselves out of the vcredist installer*, and thereby avoid the Wine errors concerning Apartment Threading when trying to install normally, the .dlls either need extra registering during the install process, or Wine has a problem with them regardless of the installer and needs fixing (which is also likely). Trying to use them with AviSynth+ and ffmpeg resulted in ffmpeg crashing with a C++ exception of some sort.

Granted, I was testing with Visual Studio 2013 builds and the corresponding vcredist, but I'm sure the same things apply for 2012 (or maybe they don't and for 2012 this would actually work; I don't know).


*Methodology ('dark' is dark.exe from the WiX Toolset):
dark vcredist2013.exe -x outputpath
go into vcRuntimeMinimum_x86 and use cabextract or 7-zip on the cab files; the MFC stuff is in vcRuntimeAdditional_x86
rename the F_CENTRAL files to .dll with the proper names (trim off the F_CENTRAL prefix and arch suffix, add .dll extension)
copy dlls to Wine's windows/system32 directory.

[ultim]

Ahh, d9 is back again. So in case if you want to know what is happening with AviSynth+: A lot!

First of all, we had a very long meeting last weekend where we focused on the most pressing matters about our GSoC application. Some still need finalization and most importantly, the ideas still need to be published in a well-formulated form. Also, please welcome our GSoC team. SEt, TurboPascal7, and myself are official mentors, while Innocenat and Firesledge have offered to help us from behind the scenes during GSoC. We'll let you know more about GSoC projects for Avs+ soon.

The MT capability of AviSynth+ is coming along nicely too, and after some more pushes from me the others have starting looking at making all the internal filters MT-safe. There is also a new API that is noteworthy, because it allows lower memory usage in general (even in single-threaded scripts!) for all filters that reserve memory on construction. Filters are being modified to make use of that too. After some bug fixes the MT build seems pretty stable, but the performance is not at its top yet. Nevertheless, this weekend or shortly after that there will probably be a public test, for the interested amongst you.

Also, I've heard today that Innocenat is experimenting with a Linux port of AviSynth+. Don't expect it to come in a couple of days since he only started recently, but work has been started nonetheless.

One kind of help we could use is if somebody stepped up to maintain the website. There will be more pages to post there real soon, so the current layout won't cut it for long. Not to mention the code duplication between pages currently isn't exactly maintainer-friendly. So it'd be nice if one of you could alter it or come forward with a new design, and maybe make changes in the future when we have new content to post. So let us know if you can help here.

Cheer for us to be successfull, we'll be back with more updates later. And don't forget to star us on GitHub :)

[turbojet]

Does this mean Innocenat is done working on tivtc (http://forum.doom9.org/showthread.php?p=1655991&highlight=tivtc#post1655991)?

If so, was there no gains or something else?

[TurboPascal7]

Does this mean Innocenat is done working on tivtc (http://forum.doom9.org/showthread.php?p=1655991&highlight=tivtc#post1655991)?
No, it doesn't.

[innocenat]

Does this mean Innocenat is done working on tivtc (http://forum.doom9.org/showthread.php?p=1655991&highlight=tivtc#post1655991)?

If so, was there no gains or something else?

No, I am not.

There are times when I have free time with only Linux available, so I use that time to port.

[turbojet]

I see, thanks for the update, eagerly anticipating a faster tivtc or a wip ivtc filter that's just as capable. Maybe you've seen it but tritical's post (http://forum.doom9.org/showthread.php?p=1658871#post1658871) may help.

[Paser]

When I try to replace the AviSynth.dll in SysWOW64 with my newly compiled one, I get this error after trying to load an avs in VirtualDub:

http://puu.sh/6HLni.png

Any way to fix this?

[Stereodude]

Maybe you should post your script...

[ultim]

When I try to replace the AviSynth.dll in SysWOW64 with my newly compiled one, I get this error after trying to load an avs in VirtualDub:

http://puu.sh/6HLni.png

Any way to fix this?

The current builds definetely work with VirtualDub. Are you using the correct architecture? E.g. if you are placing the plugin in SysWOW64, you must be running 32-bit VirtualDub, and you must use the 32-bit version of AviSynth.

If all else fails, try running AviSynth+'s setup, that does the upgrade for you correctly. But given that you take the correct DLL versions, just replacing the avisynth.dll should also be enough.

[ryrynz]

Is it stable yet? I been waiting for MT build for months. I has the itch, go go go! :D

[TurboPascal7]

You can always build it yourself or join our IRC channel (#avs-plus on Rizon) if you want to test things before they make it to this forum. ;)
MT is not fully "stable" yet, but test builds will follow very soon.

[ultim]

OK, sorry that I missed the release on the weekend, but it is not easy to stay on time with lots of things going on, like overtime and deadlines at work, private server upgrade (which must also be done on deadline or else I'll loose my data), internet outage, lot's of GSoC preparations, and even (yes) some coding for AviSynth, to name a few from the top of my head.

But I bring you now a shiny new test build, r1689, which - in contrast to the previous one - is actually usable. Probably even much better than that. So I strongly suggest everyone to give it a shot, aside that I can probably improve on the thread scheduler for some more performance, you'll have fun with it (in a good way). Just make sure you set the correct MT mode for your filters. Here (https://pad.riseup.net/p/avs_plus_mt_modes) is a snippet that you can start with, but please add some new filters to that list on your own too. Don't bother with built-in filters though, they're already handled internally, so you only need to add filters from external plugins.

The MT branch is now also the main branch of AviSynth+, which means all (even non-MT) improvements end up here, and it will be merged into "master" as soon as it has received enough testing. But now you might wonder what are the "other" user-visible changes compared to the stable release. Mainly:

innocenat has worked more on the resizers, which are now even faster and require less memory, especially (but not only) when working on planar video and you have SSE3.
This (http://forum.doom9.org/showthread.php?p=1662595#post1662595) issue is fixed, which sometimes caused that a filter function from the wrong DLL would get used. Thx for reporting the issue, real.finder.
If a plugin DLL cannot be loaded, a human-readable error from Windows is also displayed, giving the user a clue what is wrong.
Filters that reserve memory on construction can now share that same piece of memory between multiple instances, giving large memory savings in many cases. Most affected internal filters have been updated to make use of this capability, most by tp7. Some external filters will follow when the API is officially stable.
The new caching system is done, and it will result in noticably lower memory usage than any previous "classical" AviSynth version. Give it a try, you'll be surprised how much memory it brings in complex scripts.
SSE2 has been a requirement for Avs+ by mistake, this is fixed now, and you now only need an SSE-machine.

Of course all the above is paired with a lot of rewrites, refactorings, and cleanups. And then there's MT. :)

"Yeah-yeah-yeah, enough with the talk, just give me a build already." Right, http://goo.gl/e0VFYn. And don't forget to add some filters (after testing them) yourself to this list: https://pad.riseup.net/p/avs_plus_mt_modes

In my next post I'll briefly summarize how to use AviSynth+ in its multithreaded mode.

[ultim]

So, how to use MT in AviSynth+? Most of it has been posted earlier actually, but let me summarize it.

By default, your script will run in single-threaded mode, just like with SEt's build. Also, just like in SEt's build, you'll have to make sure that filters use the correct MT mode, or else they might wreak havoc. There are three MT modes (1,2,3), and they are the same modes as in (yeah you guessed correctly) SEt's build. Which means you can use the same modes that you have used with AviSynth-MT.

There are some things though that are different and/or new in AviSynth+. The first difference is *how* you set the MT mode. In AviSynth-MT, you had to use SetMTMode(X), which caused all filters following that line to use mode X (until the next call to SetMTMode()). This meant if you needed to use multiple MT modes, you had to insert all those calls in the middle of your script, littered over many places.

Setting MT modes
AviSynth+ does it differently. In AviSynth+, you specify the MT-mode for only specific filters, and those filters will then automatically use their own mode, even if there were other MT-modes inbetween. This means you can specify all the MT modes at the beginning without polluting your script. You can even make a SetMTMode.avsi if you wish and let it autoload for all of your scripts, or import() it from their top. This is much cleaner, and it allows you to maintain all your MT-modes centrally at a single place. To make this distinction clear from AviSynth+, SetMTMode() is called SetFilterMTMode() in AviSynth+.

Enabling MT
The other difference is how you actually enable multithreading. Calling SetFilterMTMode() is not enough, it sets the MT mode, but the MT mode only has an effect if MT is enabled at all. Note this means you can safely include/import/autoload your SetFilterMTMode() calls in even single-threaded scripts, and they will not be messed up. Uhm, onto the point: You enable MT by placing a single call to Prefetch(X) at the *end* of your script, where X is the number of threads to use.

Example
# This line causes all filters that don't have an MT mode explicitly use mode 2 by default.
# Mode 2 is a relatively safe choice until you don't know most of your calls to be either mode 1 or 3.
# Compared with mode 1, mode 2 trades memory for MT-safety, but only a select few filters will work with mode 1.
SetFilterMTMode("DEFAULT_MT_MODE", 2)

# FFVideoSource(), like most source filters, needs MT mode 3
SetFilterMTMode("FFVideoSource", 3)

# Now comes your script as usual
FFVideoSource(...)
trim(...)
MCTemporalDenoise(...)
...

# Enable MT!
Prefetch(4)

Closing notes (don't skip!)

Remember that MT is only stable as long as you have specified a correct MT mode for all filters.
Instead of the numbers 1-2-3, you can also use symbolic names for MT modes: MT_NICE_FILTER (1), MT_MULTI_INSTANCE (2), MT_SERIALIZED (3)
Mode 3 is evil. It is necessary for some filters, and it is usually no problem for source filters, but it can literally completely negate all advantages of MT, if such a filter is placed near the end of your script. Let us know if you meet a non-source mode 3 filter, we might be able to do something about it, but in general, avoid such calls if you want performance. (And of course, insert what you have found into here (https://pad.riseup.net/p/avs_plus_mt_modes).)
The new caches will save you a lot of memory in single-threaded scripts, but due to the way they work, they will also use more memory than before with MT enabled. The memory usage will scale much closer with the number of threads you have. Just something to keep in mind.
MT-enabled AviSynth+ triggers a latent bug in AvsPmod. Until a new version of AvsPmod is officially released, use this (http://files.line0.in/builds/AvsPmod-2.5.1-r426-x86-04874ed.7z) build. A thousand thanks to vdcrim for the fix.
Using too many threads can easily hurt performance a lot, because there are other bottlenecks too in your PC than just the CPU. For example, if you have a quad-core machine with 8 logical cores, less than 8 threads will often work much better than 8 or more.

[easyfab]

Thanks Ultim and avs+ team for this release, here my first little test on an intel I2600K.

http://pastebin.com/Di0w2Y0y

Not bad for a first version :) ,
for info avs+ single thread -> FPS (min | max | average): 28.47 | 32.69 | 30.88

The difference I see is in frame count 145 vs 103, could it explain the difference in speed ?

[ultim]

The old masktools2-a48 has a serious issue that will make things blow up/crash if MT is used. Newer rebuilds of that same plugin work correctly, such as the one from QTGMC's Modded Plugin Package (http://forum.doom9.org/showthread.php?t=156028), or TurboPascal7's rewrite (http://forum.doom9.org/showthread.php?t=169832).

[easyfab]

I have mt_masktools-26.dll (from 06_taro ) and I try with TurboPascal7's rewrite It give me the ~ same result -> FPS (min | max | average): 33.20 | 144.55 | 74.46

[ultim]

Thanks Ultim and avs+ team for this release, here my first little test on an intel I2600K.

http://pastebin.com/Di0w2Y0y

Not bad for a first version :) ,
for info avs+ single thread -> FPS (min | max | average): 28.47 | 32.69 | 30.88

The difference I see is in frame count 145 vs 103, could it explain the difference in speed ?

Hi, thanks for the test. You can also try with a bit fewer threads (5-6), it will often result in better speeds. Otherwise, as noted above, you can expect further performance improvements from MT in the future, as I'm still working on it. This was just an early test build.

As for the thread count, I'm not sure to be honest. Probably because a mode 2 plugin has internal threading too. Both are way too high though, but I'm glad to see that Avs+ has the less threads.

[easyfab]

Ultim,

You're right about threads:

threads : Prefetch(8) vs Prefetch(5)
fps average : 76.77 vs 81.52
And less memory 520 vs 444

[Groucho2004]

As for the thread count, I'm not sure to be honest. Probably because a mode 2 plugin has internal threading too. Both are way too high though, but I'm glad to see that Avs+ has the less threads.
The 145 threads from the first example are easily explained:

setmtmode(5,8) #8 threads for MT

QTGMC(preset="fast", EdiThreads=8).selecteven() #8 Edithreads * 8 MT threads = 64 threads

distributor() #Distributor is already called in AVSMeter so calling it twice makes 64 * 2 = 128 threads

Finally, add some threads for other filters and you have your 145 threads.

[Groucho2004]

@ultim
Before I start digging through the AVS+ code - How do I determine the MT mode in Avisynth+?
In AVSMeter, I have been doing it like this so far (for SEt's MT DLL), pretty much the same as it's done in x264:
AVSValue AVS_temp;
int iMTMode = 0;
try
{
AVS_temp = AVS_env->Invoke("GetMTMode", false);
iMTMode = AVS_temp.IsInt() ? AVS_temp.AsInt() : 0;
if ((iMTMode > 0) && (iMTMode < 5) && settings.bInvokeDistributor)
AVS_main = AVS_env->Invoke("Distributor", AVS_main);
}
catch (IScriptEnvironment::NotFound)
{
}


Also, is Prefetch() similar to the Distributor() call?

[qyot27]

I've run into an issue with resampling audio.

As sometimes happens, if you use AssumeFPS's sync_audio parameter, the resultant sample rate gets wonky, requiring a follow-up use of SSRC to correct it. Even worse, depending on the source and output samplerates, you have to use SSRC twice - once before the AssumeFPS, and again afterward. The canonical example would be something like 44.1kHz synced to 30000/1000 fps, resampled to 48kHz synced to 30000/1001 fps.

In 2.6a5 and r1576, this worked like it always does. But somewhere between r1576 and r1636 (r1689 also shows it), possibly in the MT or cache changes, the above operation can cause an error at the end of encoding and audio truncation when using FFAudioSource as the input source. Now, I'm not entirely sure if it's something that needs to be fixed on FFMS2's side or not, but the problem is as follows:

Script Input -> Script Output FFmpeg log message
44.1kHz -> 44.1kHz no error
44.1kHz -> 48kHz [avisynth @ 03922860] FFAudioSource: Out of bounds audio samples requested (single use of SSRC, no fps change)
44.1kHz@30fps -> 48kHz@29.97fps same error, plus truncated* (SSRC->AssumeFPS w/sync_audio=true->SSRC)
48kHz -> 48kHz no error

*IIRC, 44.1->48 without an fps change might have also had some truncation, but it wasn't nearly as bad as the resample+fps change test was.

r1689 = error, truncated
r1636 = error, truncated
r1576 = works, no truncation
2.6a5 = works, no truncation

Script:
FFmpegSource2("[AKROSS Con 2013] Okami - Quiet.mp4",atrack=-1,fpsnum=30000,fpsden=1000)
SSRC(48000,fast=false)
AssumeFPS(29.97,sync_audio=true)
SSRC(48000,fast=false)

FFmpeg:
ffmpeg -i test.avs -vn -acodec pcm_s16le test-48re.wav

This was all single-threaded. I wasn't using any MT-related functions at all (not to mention that I'm using a single-core CPU without hyperthreading).

Saving a resampled (but not fps-resynced) PCM file and then importing that in with WAVSource before doing the AssumeFPS and SSRC adjustment results in no error and possibly no truncation (or the much more minor one I mentioned above).

[ultim]

How do I determine the MT mode in Avisynth+?

You cannot get the MT mode in Avs+. IMHO it wouldn't make too much sense because an MT-mode is always associated with a filter. So at best you could ask which MT mode is a particular filter configured to use. You can ask for the number of threads though, but that is part of IScriptEnv2, which is not officially stable yet.

Also, is Prefetch() similar to the Distributor() call?

Yes, I think it is (given that I undertsand Distributor() halfway correctly). But Prefetch() shouldn't be inserted automatically, it should always be added by the user when they want a multithreaded chain. All in all, I don't think Avs+MT should need special handling in tools like Avs-MT did with the Distributor().

[Groucho2004]

You cannot get the MT mode in Avs+. IMHO it wouldn't make too much sense because an MT-mode is always associated with a filter. So at best you could ask which MT mode is a particular filter configured to use. You can ask for the number of threads though, but that is part of IScriptEnv2, which is not officially stable yet.

Thanks. I basically just want to know if a script is running multi-threaded or not.

[ultim]

I've run into an issue with resampling audio.

...


Hi qyot27,

Could you please check these special builds for you if you experience the same problem with either of them? And if so, which one?
EDIT: lol, sry i forgot to post the link: https://mega.co.nz/#F!sZkDjKRD!ZZz7oQvYP6FQ0F0rAeYOoQ

[ryrynz]

Just wondering why this is crashing my media player am using ffdshow raw.

SetFilterMTMode("", 2)
SetFilterMTMode("ffdshow_source", 3)
ffdshow_source()
Psharpen()
Prefetch(4)

I've tried stating Psharpen in a SetFilterMTMode as well but same deal. I just copied the Avisynth DLL into my SYSWOW64 folder. What am I missing?
Also is SetMemoryMax still something that should be set?

[ultim]

Thanks. I basically just want to know if a script is running multi-threaded or not.

Yes, you can tell that using IScriptEnvironment2::GetProperty(AEP_FILTERCHAIN_THREADS), but please don't use it before the whole IScriptEnvironment2 is finalized. This is not a licensing restriction, it has to do with the fact that if you start using it and the class changes after that, your application will break.

[ultim]

Just wondering why this is crashing my media player am using ffdshow raw.

SetFilterMTMode("", 2)
SetFilterMTMode("ffdshow_source", 3)
ffdshow_source()
Psharpen()
Prefetch(4)

I've tried stating Psharpen in a SetFilterMTMode as well but same deal. I just copied the Avisynth DLL into my SYSWOW64 folder. What am I missing?
Also is SetMemoryMax still something that should be set?

Hi,

I cannot try it right now because I don't have the necessary stuff installed. Does it work if you use SetFilterMTMode("", 3) instead?

[ryrynz]

It crashes even if the chain is left completely blank, which of course is no problem with 2.6 MT. Something's broken.

[TurboPascal7]

It crashes even if the chain is left completely blank, which of course is no problem with 2.6 MT. Something's broken.

Works fine (http://i3.minus.com/ibuBF0OZAJTvsH.jpg) here. Please ensure that you have correct x86 avisynth.dll in your SysWOW64 folder and x64 dll in System32 folder.
Also, are you using x64 ffdshow/mpc by any chance?

[ryrynz]

Yeah correct x86 version downloaded in SysWOW64, copied the x64 version to System32 as you said above (not required is it?) Am using MPC-HC/BE/ffdshow x86.
2.6 MT is working fine, just copied the 2.6MT version over the AVS+ version and BE/HC doesn't crash at all, I have no idea what's going on.

[turbojet]

Latest version resizer's are a bit faster but still about 2% slower than avisynth 2.60, it used to be 4% slower on amd fx8320.

[Nevilne]

divide=1 or divide=2 doesn't work in MAnalyse with avisynth+

super = MSuper()
backward_vectors = MAnalyse(super, isb = true)
forward_vectors = MAnalyse(super, isb = false)
MFlowBlur(super, backward_vectors, forward_vectors, blur=15)
working

super = MSuper()
backward_vectors = MAnalyse(super, divide=1, isb = true)
forward_vectors = MAnalyse(super, divide=1, isb = false)
MFlowBlur(super, backward_vectors, forward_vectors, blur=15)
black screen

[innocenat]

Latest version resizer's are a bit faster but still about 2% slower than avisynth 2.60, it used to be 4% slower on amd fx8320.

Can you also provide raw fps number? Also, what kernel did you use? As stated before, none of dev have AMD machine, so it isn't tested.

[qyot27]

Hi qyot27,

Could you please check these special builds for you if you experience the same problem with either of them? And if so, which one?
EDIT: lol, sry i forgot to post the link: https://mega.co.nz/#F!sZkDjKRD!ZZz7oQvYP6FQ0F0rAeYOoQ
Both builds crash immediately due to a SIGILL.

On a different computer (that has SSE2), though:
AviSynth_check.dll = error, truncation
AviSynth_nocheck.dll = error, truncation

[naoan]

Thanks for the build ultim, haven't tried much but fft3dgpu need mode 3 to operate without glitch.

[turbojet]

blankclip(1630,1920,1080,"YV12",23.976)
resize(1280,720)
measured with avsmeter 1.7.6

AVS+ 2.60
Lanczos3 116 132
Spline64 108 116
Bicubic 192 159
Bilinear 205 225
Bicubic being faster while the others are slower might be interesting.

[kypec]

... new test build[/B], r1689

Right, http://goo.gl/e0VFYn.

Can anyone be so kind and tell me how am I supposed to download comfortably from that MEGA folder? When I choose "Download as ZIP" it always got stuck at 10~13% in Firefox. When I choose standard download then I have to confirm manually each single file to be downloaded and folder hierarchy is lost, x86 & x64 files are mixed due to having identical names :angry: Not very user friendly and very error-prone IMO.

EDIT: I just tried with Chrome on Windows 7 - same issue with "Download as ZIP". :(

[ultim]

divide=1 or divide=2 doesn't work in MAnalyse with avisynth+

super = MSuper()
backward_vectors = MAnalyse(super, isb = true)
forward_vectors = MAnalyse(super, isb = false)
MFlowBlur(super, backward_vectors, forward_vectors, blur=15)
working

super = MSuper()
backward_vectors = MAnalyse(super, divide=1, isb = true)
forward_vectors = MAnalyse(super, divide=1, isb = false)
MFlowBlur(super, backward_vectors, forward_vectors, blur=15)
black screen

Will check at home.

[ultim]

Both builds crash immediately due to a SIGILL.

On a different computer (that has SSE2), though...
Ah yes, I forgot to mention this in the changelist, I'll go and edit shortly. Previous stable Avs+ have been compiled to SSE2 unknowingly, this has been already discovered and is fixed in the current MT build. SSE is still required though.

AviSynth_check.dll = error, truncation
AviSynth_nocheck.dll = error, truncation

I was afraid of this. Then I think this is something to be fixed in FFAudioSource(). The special test builds for you were the same as stable r1576 (which you said was still working), except I removed the audio caches. No changes, just pure code removal, so that the Cache filters becomes completely transparent for audio and call through directly to the source filter. So unless the audio cache has been doing something funky, I think this might be a problem in FFAudioSource(). I'm not 100% sure though, as I'm not familiar with the audio cache of old AviSynth.

[ultim]

Thanks for the build ultim, haven't tried much but fft3dgpu need mode 3 to operate without glitch.

Thx. This will probably be true for all filters that use the GPU. For these filters, there is nothing Avs+ can do about them, aside from using mode 3. Try to place these calls towards the beginning of your script to minimize the performance affect on other filters.

[zero9999]

Can anyone be so kind and tell me how am I supposed to download comfortably from that MEGA folder? When I choose "Download as ZIP" it always got stuck at 10~13% in Firefox. When I choose standard download then I have to confirm manually each single file to be downloaded and folder hierarchy is lost, x86 & x64 files are mixed due to having identical names :angry: Not very user friendly and very error-prone IMO.

EDIT: I just tried with Chrome on Windows 7 - same issue with "Download as ZIP". :(
AviSynth+-2.6.0.5-MT-r1689-g0d5dfb7.7z (https://files.line0.in/builds/AviSynth%2B-2.6.0.5-MT-r1689-g0d5dfb7.7z)
AvsPmod-2.5.1-r426-x86-04874ed.7z (https://files.line0.in/builds/AvsPmod-2.5.1-r426-x86-04874ed.7z)

[Groucho2004]

blankclip(1630,1920,1080,"YV12",23.976)
resize(1280,720)
measured with avsmeter 1.7.6

AVS+ 2.60
Lanczos3 116 132
Spline64 108 116
Bicubic 192 159
Bilinear 205 225
Bicubic being faster while the others are slower might be interesting.

Same script with i5 2500K @ 4GHz:
(There is no Lanczos3 so I left it out)
AVS+ r1689 2.60A5
Spline64 161 190
Bicubic 341 298
Bilinear 382 461

The relative differences seem roughly the same for Intel/AMD, Intel is just a lot faster.

[innocenat]

turbojet, Groucho2004: Would you guys also mind testing horizontal resizer only? (i.e. 1920x1080 to 1280x1080 etc.) Thank you in advance.

[Groucho2004]

turbojet, Groucho2004: Would you guys also mind testing horizontal resizer only? (i.e. 1920x1080 to 1280x1080 etc.) Thank you in advance.

blankclip(1630,1920,1080,"YV12",23.976)

AVS+ 2.60A5
Spline64resize(1280,1080) 211 250
Bicubicresize(1280,1080) 472 379
Bilinearresize(1280,1080) 472 571

[innocenat]

blankclip(1630,1920,1080,"YV12",23.976)

AVS+ 2.60A5
Spline64resize(1280,1080) 211 250
Bicubicresize(1280,1080) 472 379
Bilinearresize(1280,1080) 472 571

Thank you very much. It seems that it is not as fast as I initially tested and expected.

[Nevilne]

divide=1 or divide=2 doesn't work in MAnalyse with avisynth+

Will check at home.

Thanks


Thanks for the build ultim, haven't tried much but fft3dgpu need mode 3 to operate without glitch.

Thx. This will probably be true for all filters that use the GPU. For these filters, there is nothing Avs+ can do about them, aside from using mode 3. Try to place these calls towards the beginning of your script to minimize the performance affect on other filters.

But should there be such a huge performance hit? I have a heavy script which has setmtmodes 2,3 and 4 for fft3dgpu at the end, it runs at 9 fps and fft3dgpu doesn't slow it down at all.

avs+ runs it at 12 fps with mode 2 but if i choose mode3 for fft3dgpu it runs at 6 fps.
However, if i put prefetch before fft3dgpu script seems to run correctly, and at 16 fps.

[innocenat]

Just a note: having filter after Prefetch() will currently result in two group of pipelined filter: filter before Prefetch() will run in number of threads specify, while filter after Prefetch() will run in single, main thread. Not sure if this behaviour will change or not.

[qyot27]

Ah yes, I forgot to mention this in the changelist, I'll go and edit shortly. Previous stable Avs+ have been compiled to SSE2 unknowingly, this has been already discovered and is fixed in the current MT build. SSE is still required though.
Yeah, for my own builds with VS2013, I just disabled additional CPU optimizations completely (/arch:IA32), regardless of the fact that I do have an SSE-capable processor.

How much of an equivalent is /arch in MSVC to GCC's -msse/etc. parameters? Does it have an impact on the intrinsics (since the option to dis/enable them is different), or are the two completely separate and the /arch stuff just optimizes the C/C++ parts?

I was afraid of this. Then I think this is something to be fixed in FFAudioSource(). The special test builds for you were the same as stable r1576 (which you said was still working), except I removed the audio caches. No changes, just pure code removal, so that the Cache filters becomes completely transparent for audio and call through directly to the source filter. So unless the audio cache has been doing something funky, I think this might be a problem in FFAudioSource(). I'm not 100% sure though, as I'm not familiar with the audio cache of old AviSynth.
I'll have to see. From looking at FFMS2's audiosource.cpp, that error message is defined in the FFMS_AudioSource::GetAudio function, here:
https://github.com/FFMS/ffms2/blob/master/src/core/audiosource.cpp#L343

Not sure if anything particular jumps out, but I thought I'd still point to it.

[innocenat]

How much of an equivalent is /arch in MSVC to GCC's -msse/etc. parameters? Does it have an impact on the intrinsics (since the option to dis/enable them is different), or are the two completely separate and the /arch stuff just optimizes the C/C++ parts?


It only affect compiler-generated code. Unlike GCC/Clang, the flag has no effect on intrinsics. But we have dynamic dispatcher with pure-C path for every internal filter so multiple version of filter is chosen automatically at runtime.

[ultim]

But should there be such a huge performance hit? I have a heavy script which has setmtmodes 2,3 and 4 for fft3dgpu at the end, it runs at 9 fps and fft3dgpu doesn't slow it down at all.

avs+ runs it at 12 fps with mode 2 but if i choose mode3 for fft3dgpu it runs at 6 fps.
However, if i put prefetch before fft3dgpu script seems to run correctly, and at 16 fps.

Yes, everything that you showed here is nothing out of ordinary. If you put a mode 3 fiilter in your script, all filters before it will also be serialized, which is the cause of the performance hit. This is why I recommended earlier that if you have a mode 3 filter, try to put it towards the beginning, which minimizes this hit.

The solution that you have implemneted, putting fft3dgpu *after* Prefetch(), is also a perfect solution, and is even better than placing it just before the Prefetch() if you have nothing after fft3dgpu.

[ultim]

Yeah, for my own builds with VS2013, I just disabled additional CPU optimizations completely (/arch:IA32), regardless of the fact that I do have an SSE-capable processor.

How much of an equivalent is /arch in MSVC to GCC's -msse/etc. parameters? Does it have an impact on the intrinsics (since the option to dis/enable them is different), or are the two completely separate and the /arch stuff just optimizes the C/C++ parts?


I'll have to see. From looking at FFMS2's audiosource.cpp, that error message is defined in the FFMS_AudioSource::GetAudio function, here:
https://github.com/FFMS/ffms2/blob/master/src/core/audiosource.cpp#L343

Not sure if anything particular jumps out, but I thought I'd still point to it.

Oh, you get that error? Then I'm puzzled and even less sure than I was before. This definetely needs investigation.

[qyot27]

It only affect compiler-generated code. Unlike GCC/Clang, the flag has no effect on intrinsics.
Ah, I see.

Oh, you get that error? Then I'm puzzled and even less sure than I was before. This definetely needs investigation.
Correct. I did a little bit more testing, and it happens with both SSRC and ResampleAudio, with slightly different behavior (see below). Based on the wording of the error message, could it be that the resampling filters are requesting extra samples out of FFAudioSource that just aren't there, leading to the error?




The results are as follows:
SSRC = error message is thrown just before the end of the read, script exits immediately causing audio to be truncated at the end

ResampleAudio = error message is thrown immediately, script exits, empty output file

No resampling step(s) at all = output audio is 44056kHz, no error is thrown, read ends successfully with no truncation


Steps to reproduce:
Generate test video+audio with FFmpeg:
ffmpeg -f lavfi -i testsrc=duration=60:size=352x176:rate=30 -f lavfi -i aevalsrc="sin(440*2*PI*t):s=44100" -vcodec mpeg4 -acodec aac -strict experimental -t 60 output.mp4

Index the file:
ffmsindex -t -1 output.mp4

Script:
FFmpegSource2("output.mp4",atrack=-1,fpsnum=30000,fpsden=1000)
#SSRC(48000,fast=false)
AssumeFPS(29.97,sync_audio=true)
#SSRC(48000,fast=false)
#ResampleAudio(48000)
Uncomment for SSRC vs. ResampleAudio tests.

[turbojet]

What I meant by Lanczos3 is lanczosresize(taps=3) which is default.

Interesting that intel has the same kind of results, I don't have time to test horizontally only right now but I can do it in a few hours if it's still needed.

[kypec]

AviSynth+-2.6.0.5-MT-r1689-g0d5dfb7.7z (https://files.line0.in/builds/AviSynth%2B-2.6.0.5-MT-r1689-g0d5dfb7.7z)
AvsPmod-2.5.1-r426-x86-04874ed.7z (https://files.line0.in/builds/AvsPmod-2.5.1-r426-x86-04874ed.7z)
:thanks: although you might consider to replace secure https linking with standard non-secure http due to untrusted (self-signed) certificate issues for potential downloaders of your builds.

[naoan]

Thx. This will probably be true for all filters that use the GPU. For these filters, there is nothing Avs+ can do about them, aside from using mode 3. Try to place these calls towards the beginning of your script to minimize the performance affect on other filters.

Yeah it's like mode 5 in the old mt. Thanks for the tips but I need it last, and it's actually fast enough on my simple real time ffdshow-avisynth+ anyway (just it and fastlinedarkenmod).

[ultim]

Yeah it's like mode 5 in the old mt. Thanks for the tips but I need it last, and it's actually fast enough on my simple real time ffdshow-avisynth+ anyway (just it and fastlinedarkenmod).

If it needs to be last, you can also place it after the call to Prefetch().

[naoan]

If it needs to be last, you can also place it after the call to Prefetch().

Nice! didn't know you could do it like that, seems a bit faster/less frame drop now. Thanks! :D

[kypec]

AviSynth+-2.6.0.5-MT-r1689-g0d5dfb7.7z (https://files.line0.in/builds/AviSynth%2B-2.6.0.5-MT-r1689-g0d5dfb7.7z)
AvsPmod-2.5.1-r426-x86-04874ed.7z (https://files.line0.in/builds/AvsPmod-2.5.1-r426-x86-04874ed.7z)

I've replaced my Avisynth installation with your x86 build and also replaced AvsPmod files but now AvsPmod refuses to even start with error message below:
http://i57.tinypic.com/23ubvx1.jpg
Clicking "Yes" and navigating to C:\Windows\SysWOW64 where that Avisynth+ DLL resides does not work either. :(

[ryrynz]

Hoping for a build that works with ffdshow, tried it on both my computers win8/win7 i7/i5 and both crash instantly, both are working fine with 2.6MT.

[turbojet]

blankclip(1630,1920,1080,"YV12",23.976)
resize(1280,1080) AVS+ 2.6
Lanczos 139 153
Spline64 142 135
Bicubic 241 175
Bilinear 246 240


Same conclusion as Groucho2004's test with lanczos included.

[ultim]

So here are some more news I've promised. The list of our Google Summer of Code ideas is online! It is available from the homepage (http://avs-plus.net) with a single click, but here is a direct link (http://www.avs-plus.net/gsoc-ideas.php) for you. You'll find all kinds of interesting projects there, from native GPU support to scripting language improvements and a GUI plugin manager, just to name a few. Now, cross your fingers for AviSynth to be accepted.

[Stereodude]

Your direct link gives a 404.

I also have a question about AviSynth+. I see a big push related to MT in the past week. I thought the whole idea was to get better performance without requiring the user to mess around with MT commands and the like? I'm probably put off to the idea because my experience with MT support in AviSynth is that it's a complete disaster that barely works even on the rare occasions when the stars align just right and I'd not rather not have those headaches again. I've had much better luck with MP_Pipeline at speeding up scripts than I ever had with MT and it doesn't require any sort of celestial alignment or blood sacrifices.

[ultim]

Your direct link gives a 404.
Works for me.


I also have a question about AviSynth+. I see a big push related to MT in the past week. I thought the whole idea was to get better performance without requiring the user to mess around with MT commands and the like?
Ultimately, that is what everybody (including us developers) would like to see. But unfortunately what you wish for is simply technically not possible for many filters due to the way they work. What we can do though is to minimize the MT-related commands the user has to issue, instead of completely eliminating them. Minimizing the MT-commands for script writers is the collective goal of many of our efforts, such as many filter rewrites (both internal and external), adding support for filters to signal to Avs+ the MT mode they need, the way SetFilterMTMode() works, and even of this (http://forum.doom9.org/showpost.php?p=1649886&postcount=181) post to teach plugin developers how to support MT hassle-free for the user.

So we do what we can, but sometimes there are hard limits to what is possible at all.

[TurboPascal7]

I thought the whole idea was to get better performance without requiring the user to mess around with MT commands and the like?
That's right. When MT is finalized, the only thing the user will have to write in his script is the Prefetch call, which later can probably be added by programs like avsmeter, avs2yuv and the likes. All SetFilterMtMode calls will be hidden in a single .avsi script.

Compared to all those ### MT_Pipeline requires you to write, this is nothing.

Now, the problem is to build this avsi file with MT modes definitions. I put this pad (https://pad.riseup.net/p/avs_plus_mt_modes) about two weeks ago and posted the link on our IRC channel, ultim also linked it here a few days ago. But since the initial script, only one line has been added (fft3dgpu). With this kind of activity you can't expect MT to get finalized and become user-side overhead free any time soon.

[HeadlessCow]

That's right. When MT is finalized, the only thing the user will have to write in his script is the Prefetch call, which later can probably be added by programs like avsmeter, avs2yuv and the likes. All SetFilterMtMode calls will be hidden in a single .avsi script.

Compared to all those ### MT_Pipeline requires you to write, this is nothing.

Now, the problem is to build this avsi file with MT modes definitions. I put this pad (https://pad.riseup.net/p/avs_plus_mt_modes) about two weeks ago and posted the link on our IRC channel, ultim also linked it here a few days ago. But since the initial script, only one line has been added (fft3dgpu). With this kind of activity you can't expect MT to get finalized and become user-side overhead free any time soon.

Is there a reason why you couldn't require the filters to implement a new method that returns the MT modes that they support rather than relying on a community generated avsi script? You'd still need the script for older plugins, but anything new (or anything that you're updating) would just automatically work.

[TurboPascal7]

Is there a reason why you couldn't require the filters to implement a new method that returns the MT modes that they support rather than relying on a community generated avsi script? You'd still need the script for older plugins, but anything new (or anything that you're updating) would just automatically work.
No, there is no reason we couldn't do this. That's why we did. :) It works by abusing SetCacheHints (https://github.com/AviSynth/AviSynthPlus/blob/0d5dfb39f9b45a268884509f77022f604a14f4a7/avs_core/filters/transform.h#L63) right now but allows keeping backward compatibility.

[HeadlessCow]

No, there is no reason we couldn't do this. That's why we did. :) It works by abusing SetCacheHints (https://github.com/AviSynth/AviSynthPlus/blob/0d5dfb39f9b45a268884509f77022f604a14f4a7/avs_core/filters/transform.h#L63) right now but allows keeping backward compatibility.

Ah, perfect (and sneaky)! I hadn't seen mention of it in the thread, so I just wanted to make sure the obvious option hadn't been accidentally overlooked :-D

[turbojet]

Here's some results with fastest mt mode:

tfm 1
tdecimate 3 or off, 1 fails
telecide 1
decimate 3 or off, 1 and 2 fails
fielddeinterlace 2, 1 fails
dss2 1, 2 is very slow
lwlibavvideosource(dr=true) 2 is 25% faster with much higher cpu usage, 75 vs 17%, 1 fails
nnedi3 2, 1 corruption
colormatrix("Rec.709->Rec.601") 2 is <1% faster than all modes or off
dfttest 1
fdecimate 1 and 2 tie
it 3, 1 fails, 2 is same speed as 3 but 4x cpu load
jinc36resize(3840,2160) 2, 1 is corrupt
leakkerneldeint(1) 1
repal 3, 1 fails, 2 is slow
sorathread 3, 1 and 2 fails
srestore all tie
tdeint 1
unblend 1 or 2
uncomb 3 or off
unsharphq 1
yadif 1
yadifmod(edeint=nnedi3()) 2, 1 fails

All defaults unless (). When testing it's best to use a real source, some filters work on blankclip() but not on real sources
with the same mtmode.

Mode 1 works on quite a few things. One thing I didn't consider (much) is cpu load. I used prefetch(4) which 50% with most filters while mode 3 and off used 12%. mtmodes during realtime watching should be a big benefit but unless slow filters are used there's a little bit of a slowdown when x264 encoding.

[TurboPascal7]

Please do check if the actual output is correct. Fast but broken mode is useless.
I'm asking because for example nnedi3 with mode 1 is completely broken yet you specified it. I'm pretty sure that's the case with most filters in the list.
Easy way of checking would be using something like ColorBars(1920, 1080, "YV12").addgrainc(10000, 10000, seed=1) as a source filter. It doesn't always work right but will do for most stuff (like needi3).

[turbojet]

Does the corruption like nnedi3 only happen with mode 1?

[TurboPascal7]

Does the corruption like nnedi3 only happen with mode 1?
It most cases - yes.

Nnedi3 uses some buffers to do its dirty work and with mode 1 you get multiple threads writing data from different frames to the same buffer. This causes corruption when later someone tries to read from this buffer and gets not what was expected. Most of "more complicated" filters use some kind of temporary storage thus won't work well with this mode. Simple filters might.

Mode 2 doesn't have this issue because multiple threads will get their own buffers and no data will be shared. Hence mode 2 is the "default" mode which should work with most filters, but it wastes memory like crazy (take SangNom2 for example - for 1080p YV12 frame, size of temporary buffers is about 10MB, so with 4 threads you get 40MBs on single filter invocation. Now add some usual supersampling to this and multiple invocations in most aa scripts and... you get the idea).

If the filter requires sequential access or uses some global storage (i.e. written by a very bad person), then mode 3 is the only way to go.

[turbojet]

Tested all the reported 1's and other than nnedi3, jinc36resize didn't look correct. Others worked fine.

[ultim]

Ah, perfect (and sneaky)! I hadn't seen mention of it in the thread, so I just wanted to make sure the obvious option hadn't been accidentally overlooked :-D

It wasn't mentioned to avoid plugin writers accidentially using those parts of the new interface which are not safe. The MT-mode specification is safe, but be sure not to use IScriptEnvironment2 at all, for exmaple.

The good thing about MT-mode specification as shown by TurboPascal7, as he already noted, is that it keeps compatibility. So plugins can implemnent it without breaking older AviSynth versions, or non-plus AviSynth. Putting it into SetCacheHints() is ugly I know (TurboPascal7 was very much against it putting it there for this reason), but it is the only way for me to let plugins report their MT-mode without breaking other AviSynths.

Anyway, long story short, you can follow the example here (https://github.com/AviSynth/AviSynthPlus/blob/0d5dfb39f9b45a268884509f77022f604a14f4a7/avs_core/filters/transform.h#L63), but don't use any other part of the new interface yet. For plugins that report the MT mode using SetCacheHint(), the user won't have to add a SetFilterMode() call.

[LaTo]

What is a good way to detect Avisynth+ without losing compatibility with regular Avisynth?
I need that to automatically disable internal multi-threading in my filters.
Thanks.

[Groucho2004]

What is the best way to detect Avisynth+ without losing compatibility with regular Avisynth?
I need that to automatically disable internal multi-threading in my filters.
Thanks.
There are several ways to identify the avisynth.dll.

1. Query "AVISYNTH_INTERFACE_VERSION". This only tells you if it's 2.5x or 2.6x

2. Parse the version string ("VersionString"). A bit unreliable.

3. Parse the export functions. That's the most useful and what I do in AVSMeter. For example:
Only 2.6x versions export "DeleteScriptEnvironment()"
Only AVS+ exports "CreateScriptEnvironment2"
Only SEt's and tsp's MT versions export "GetMTMode()".

A combination of the above should provide enough info.

[jpsdr]

What is a good way to detect Avisynth+ without losing compatibility with regular Avisynth?
I need that to automatically disable internal multi-threading in my filters.
Thanks.

Should it be better to detect if MT mode is enabled ?
If someone use Avisynth+ without using multi-threading, i think internal multi-threading should be kept.

[ryrynz]

Running Avsmeter with prefetch enabled causes the benchmark to slow down considerably.


[General info]
Log file created with: AVSMeter 1.7.6 (AVS 2.6, x86)
Avisynth version: AviSynth+ 0.1 (r1689, x86)


[Clip info]
Number of frames: 1726
Length (hhh:mm:ss.ms): 000:01:11.989
Frame width: 704
Frame height: 480
Framerate: 23.976 (2500000/104271)
Interlaced: No
Colorspace: YV12
Audio channels: 2
Audio bits/sample: 16
Audio samples/sec: 48000
Audio samples: 3454656


[Runtime info]
Frames processed: 8 (0 - 7)
FPS (min | max | average): 0.017 | 237169 | 0.044
CPU usage (average): 6%
Thread count: 27
Physical Memory usage (peak): 90 MB
Virtual Memory usage (peak): 173 MB
Time (elapsed): 000:03:00.148


[Script]
SetFilterMTMode("", 2)
SetFilterMTMode("Directshowsource", 3)
Directshowsource("c:\1.mkv")
Deblock(20)
VagueDenoiser(threshold=1.0, nsteps=8, chromaT=0)
Hysteria(strength=0.2, highthresh=200)
Prefetch(4)


[Performance data]
Frame Frames/sec Time/frame(ms) CPU(%) Threads PhysMEM(MB) VirtMEM(MB)
0 0.017 60057.016993 0 32 82 168
1 0.017 60014.952892 0 28 88 171
2 171.206 5.840931 0 28 88 171
3 8.837 113.165406 12 28 90 173
4 33203.609 0.030117 12 28 90 173
5 237168.577 0.004216 12 28 90 173
6 83.888 11.920692 12 28 90 173
7 0.017 59944.863827 0 27 90 173


Even just having deblock as the only filter with the prefetch call causes it to lag too, so it appears directly related to the prefetch call.
This script using Avisynth 2.6MT finishes in 33 seconds, avs+ without the prefetch call finishes it in 40 seconds.

[TurboPascal7]

Looks like it breaks because of the SetFilterMTMode("", 2) line. SetFilterMTMode("audiodub",3) should fix it. Apparently AudioDub (called by dss internally) has issues with MT mode 2 which you try to apply to it by default.
And I still believe this way of setting the default MT mode should be removed. Brings nothings but problems.


Should it be better to detect if MT mode is enabled ?
If someone use Avisynth+ without using multi-threading, i think internal multi-threading should be kept.
In theory - yes. But you have to use IScriptEnviroment2 to get number of threads and it's strongly discouraged right now.

The most optimal way from performance point of view would be providing a wrapper function around the plugin which would set number of threads to 1 and pass all other parameters unchanged right to the plugin. Something like:
function awesome_filter(clip c, int "thresh") {
awesome_filter(c, thresh, threads=1)
}
But this is again far from optimal because it's not currently possible to get number of threads in the script (it's not known at "compile" time) so you can't check if you're actually running MT in this function. But at least the user can comment this wrapper out if he wants to re-enable internal threading.

[Gavino]

Apparently AudioDub (called by dss internally) has issues with MT mode 2
That's strange, since AudioDub() doesn't do any real work of its own - it just redirects video and audio requests (GetFrame() and GetAudio()) to the appropriate input clip.

[aegisofrime]

I'm getting a few instances whereby my script just stops suddenly. It doesn't crash, the encoding process just stops. Perhaps signs of a deadlock?

Here's my script:


SetFilterMTMode("QTGMC",2)
SetFilterMTMode("FFVideoSource",2)
LoadPlugin("C:\Program Files (x86)\MeGUI\tools\ffms\ffms2.dll")
FFVideoSource("I:\Temp\Test.mkv", fpsnum=24000, fpsden=1001, threads=1)
Dither_convert_8_to_16()
Dither_y_gamma_to_linear()
Dither_resize16(1280,720)
Dither_y_linear_to_gamma()
DitherPost()
QTGMC(Preset="Slow",InputType=1)
InterFrame(Cores=1,GPU=True)
PreFetch(3)


I'm also seeing the same thing when QTGMC'ing interlaced DVD content. I'm using Vit's modded Masktools and RGTools in place of RemoveGrain. Is there some logging mode I can invoke to help diagnose this problem?

[Groucho2004]

[B]SetFilterMTMode("QTGMC",2)
SetFilterMTMode("FFVideoSource",2)

You really should read ultim's notes (http://forum.doom9.org/showthread.php?p=1666371#post1666371) on how to use MT.
FFVideoSource should have Mode 3. I also doubt that you can set a mode for an entire script that uses a whole bunch of filters (QTGMC) which might each need to be treated differently.

[aegisofrime]

You really should read ultim's notes (http://forum.doom9.org/showthread.php?p=1666371#post1666371) on how to use MT.
FFVideoSource should have Mode 3. I also doubt that you can set a mode for an entire script that uses a whole bunch of filters (QTGMC) which might each need to be treated differently.

Thanks, I guess I missed the part about FFVideoSource. I guess I will also need to do some trial and error for the individual filters in QTGMC?

[lansing]

can someone put the mt version on another storage site? I can't download it from mega, when I try to download it as a zip, it always stuck at 17%.

[Groucho2004]

I guess I will also need to do some trial and error for the individual filters in QTGMC?
Yes, although here (https://pad.riseup.net/p/avs_plus_mt_modes) is already some info.

[ultim]

You cannot set the MT mode on script function calls, only on binary filters. So to make QTGMC work, all filters that it uses internally need their MT mode set. I did a basic test though when the the current test build was published, and QTGMC worked fine with a single SetFilterMTMode("", 2) call at the beginning and default arguments, so that is a start. But keep in mind that non-default arguments can easily require more complex MT-mode settings.

Anyway, thanks for all the testing. A few more days and I will begin hacking around it at large again. I am also thinking it might be very useful to have a logging framework in place (and maybe start with that), it could provide useful information about MT mode settings or other warnings to script writers. There is definetely a place to improve on the performance of the current version, and be assured that I will look at that in detail, as well as the issues that have been posted.

[TurboPascal7]

can someone put the mt version on another storage site? I can't download it from mega, when I try to download it as a zip, it always stuck at 17%.

This should do (http://forum.doom9.org/showthread.php?p=1666512#post1666512).

[lansing]

I did a simple test with mpeg2source, the prefetch() switch doesn't seem to do anything. Whether i put it there, prefetch(1) for single thread or prefetch(x), mt is always on.

[TurboPascal7]

I did a simple test with mpeg2source, the prefetch() switch doesn't seem to do anything. Whether i put it there, prefetch(1) for single thread or prefetch(x), mt is always on.

Please do post your full script. Also the best way of running a script singlethreaded is not including a prefetch call at all.

[lansing]

SetFilterMTMode("MPEG2Source",3)
MPEG2Source("sample.d2v", cpu=0)


Even when I took out the setfiltermtmode line, it still ran on mt. Here's the log from avsmeter:


AVSMeter 1.7.6 (AVS 2.6, x86) by Groucho2004
AviSynth+ 0.1 (r1689, x86)

Number of frames: 36853
Length (hhh:mm:ss.ms): 000:20:29.662
Frame width: 720
Frame height: 480
Framerate: 29.970 (30000/1001)
Interlaced: No
Colorspace: YV12

Frames processed: 6180 (0 - 6179)
FPS (min | max | average): 248.7 | 340.6 | 295.5
CPU usage (average): 12%
Thread count: 13
Physical Memory usage (peak): 35 MB
Virtual Memory usage (peak): 69 MB
Time (elapsed): 000:00:20.915

Press any key to exit...

[TurboPascal7]

The fact that you see more threads doesn't mean it's running MT. It's a minor implementation detail which might change in the future or it might not.
You can see that your CPU load is still locked at 12% which means one running thread on a 8-threads system.

[lansing]

No, it's running at 12% cpu because that's the max speed mpeg2source can run in my machine on multithread. If I ran the same vob file with ffvideosource, it would clearly show that mt is on because it would be a lot faster.

ffvideosource("sample.vob", fpsnum=30000, fpsden=1001)


AVSMeter 1.7.6 (AVS 2.6, x86) by Groucho2004
AviSynth+ 0.1 (r1689, x86)

Number of frames: 36849
Length (hhh:mm:ss.ms): 000:20:29.528
Frame width: 720
Frame height: 480
Framerate: 29.970 (30000/1001)
Interlaced: No
Colorspace: YV12

Frames processed: 36849 (0 - 36848)
FPS (min | max | average): 1242 | 1818 | 1660
CPU usage (average): 65%
Thread count: 30
Physical Memory usage (peak): 52 MB
Virtual Memory usage (peak): 68 MB
Time (elapsed): 000:00:22.202

[TurboPascal7]

Yeah because ffvideosource has internal mulithreading. It even has threads parameter which you can set to one.
There is no spoon.

[lansing]

Yeah because ffvideosource has internal mulithreading. It even has threads parameter which you can set to one.
There is no spoon.

You are right. I tried ran it with tnlmeans and took out the prefetch call, then it always ran at 12% cpu. When I change it to prefetch(8), it ran at 99%.

[ryrynz]

Any ideas on what caused my slowdown with avsmeter? Can it be reproduced?

[TurboPascal7]

Any ideas on what caused my slowdown with avsmeter? Can it be reproduced?
Did you try what I posted (http://forum.doom9.org/showthread.php?p=1667851#post1667851)? First two lines are about your directshowsource problem.

[ryrynz]

That's it. Wow. Avs+ did in 3 seconds what 2.6MT took 30 seconds to do..
Can audiodub be made mtmode 2 compatible?

[ultim]

That's it. Wow. Avs+ did in 3 seconds what 2.6MT took 30 seconds to do..
Can audiodub be made mtmode 2 compatible?

Wut? Tbh that looks too good to be true. All frames delivered with correct output?

[Groucho2004]

Wut?
That's what I thought, too. I'd like to see the log.

[ryrynz]

Actually forgot to edit the script back to it's original state.. it finished in 10 seconds.

http://codepaste.net/krm7oy

[Groucho2004]

Actually forgot to edit the script back to it's original state.. it finished in 10 seconds.

http://codepaste.net/krm7oy
So, you get 160 fps with AVS+ compared to 50 fps with SEt's MT Avisynth? And a fraction of the memory usage? Very nice.

[Seedmanc]

Neither AvsPmod nor AVSmeter seem to work with AVS+ for me, both just hang up whenever I try to open a script, even a simple "version()" call. Have to use Virtualdub again.

[Groucho2004]

Neither AvsPmod nor AVSmeter seem to work with AVS+ for me, both just hang up whenever I try to open a script, even a simple "version()" call. Have to use Virtualdub again.
Elaborate on "hang up". No error message?

[TurboPascal7]

Dear google translate users, please, get out of the MT modes pad.

[ryrynz]

So, you get 160 fps with AVS+ compared to 50 fps with SEt's MT Avisynth? And a fraction of the memory usage? Very nice.

Could drop memory usage down on 2.6MT, it's hard to tell what value you can use setmemorymax at for it to be stable.. but yes, a big difference here. Makes me wonder if 2.6MT is being benchmarked correctly, Avs+ is showing an extra 7 threads.

[TurboPascal7]

Avs+ creates two thread pools - one for prefetching threads (main pool) and one for user threads - avstp replacement allowing for more efficient internal threading implementation in plugins that cannot run with frame-level threading. So avs+ having more threads is hardly surprising.
That said, amount of threads right now is a bit silly. As I see it, with prefetch(4) it should be around 1 (main) + 3 (prefetch threads so 4 in total with main) + 4 (second thread pool) = 8, and not 20 or so created right now. Of course it hardly matters since most threads are idling, but should probably get fixed as mt becomes more stable.

[Seedmanc]

Groucho2004, well, I left both overnight trying to open a script and now I see both the same way I left them. I think I enabled logging in AVSmeter, but I don't see any logs in the folder. Meter remained at "AVSMeter 1.7.6 (AVS 2.6, x86) by Groucho2004" line in the console window.

[Groucho2004]

Groucho2004, well, I left both overnight trying to open a script and now I see both the same way I left them. I think I enabled logging in AVSmeter, but I don't see any logs in the folder. Meter remained at "AVSMeter 1.7.6 (AVS 2.6, x86) by Groucho2004" line in the console window.
You don't even see the "Analysing script, please wait..." line? Post your script.

[Seedmanc]

It looks like the problem was in "avsfilternet.dll" being in my plugins folder, now that I removed it it seems to work fine. It wasn't a problem with 2.6 though.

[tormento]

Some testing done this afternoon with 1080P material.

The following script works ok:

LoadPlugin("D:\eseguibili\media\DGDecNV\DGDecodeNV.dll")
SetFilterMTMode("", 2)
SetFilterMTMode("DGSource", 3)
DGSource("E:\in\something.dgi")
#CompTest(1)
ChangeFPS(last,last,true)
SMDegrain (tr=6,PreFilter=2,thSAD=600,contrasharp=false,refinemotion=true,plane=0,chroma=false,lsb=true,mode=6)
Prefetch(3)

The following hangs in MeGui preview:

LoadPlugin("D:\eseguibili\media\DGDecNV\DGDecodeNV.dll")
SetFilterMTMode("", 2)
SetFilterMTMode("DGSource", 3)
DGSource("E:\in\something.dgi")
#CompTest(1)
ChangeFPS(last,last,true)
SMDegrain (tr=6,PreFilter=2,thSAD=600,contrasharp=false,refinemotion=true,lsb_out=true,plane=0,chroma=false,Globals=2)
SMDegrain (tr=6, thSAD=600,contrasharp=false,refinemotion=true,lsb_in =true,plane=0,chroma=false,Globals=1,mode=6)
Prefetch(3)

Both works ok under AviSynth_MT from Set, which is even a bit faster.

[lansing]

mpeg2source doesn't seem to work with avs+ mt on. I have a 1080p mpeg2 video which I indexed with dgindex, and trying to test with mt on, it gave me a green screen onload with avspmod, and when I try seek, the program just hangs. All three mt mode will hang.


SetFilterMTMode("MPEG2Source", 2)
MPEG2Source("sample.d2v", cpu=0)
Prefetch(4)

[udobroemme]

I think it was mentioned before... :)
For source filters you have to use
SetFilterMTMode("MPEG2Source", 3)

[lansing]

After a few more tests, I figure setting prefetch to 1 core will work, anything more than 1 will cause avsmeter to hang. And it doesn't matter which mode I use, all three will work if I call prefetch(1).

[Zep]

After a few more tests, I figure setting prefetch to 1 core will work, anything more than 1 will cause avsmeter to hang. And it doesn't matter which mode I use, all three will work if I call prefetch(1).

the question then, are

no prefetch in the script
prefetch(0)
prefetch(1)

truly equivalent? If so that should be why all modes worked.

yes I'm curious lol

[innocenat]

the question then, are

no prefetch in the script
prefetch(0)
prefetch(1)

truly equivalent? If so that should be why all modes worked.

yes I'm curious lol

No prefetch is technically Prefetch(0). Prefetch(1) is currently equivalent, but it is not guaranteed.

Prefetch itself predicts what frame will be requested next and fetch that frame in advance (in available thread). Prefetch(0) means to not use prefetching. Prefetch(1) will create a thread pool of one thread and do prefetching solely on that thread. Thus there is no guarantee that Prefetch(1) will request frame from its upstream filter in same order as it is requested (but it is currently this way)

[pinterf]

Tested AviSynth+ 0.1 (r1689, x86) mt for my 8mm film restoring project, while experimenting with Dither package, 10 bit x264 output, etc.

This film restoring script included Depan, RemoveDirtMC, MDegrain2, Crop16, Resize16 and converting between 8 bit and 16 bit stacked format and other functions from Dither Package.

SET's 2013.09.28 MT version was OK with SetMTMode(2). 95% CPU usage by AvsMeter.

So I generally enabled MT_MULTI_INSTANCE (except AviSource) but did not work. Tried to test MT modes but got no consistent results.
Finally turned out that either Dither_Out()/DitherPost() or Dither_Resize16() should be MT_SERIALIZED when MDeGrain2 used. When MDegrain2 was not used, even MT_NICE_FILTER worked for them. MDegrain2 comes from the modified mvtools2 package came with Dither tools.

Narrowed down the problem for this script.

For working version change comment/uncomment these lines: SetFilterMTMode("Dither_Resize16"...


# some globals for the clip
play_speed=16
trim_begin=0 #120
seconds=120
trim_end=trim_begin + play_speed*seconds
_grey=true
film="c:\Tape13\Videos\1975_KeskenyFilm.avi"

# AVS+ version
SetFilterMTMode("", MT_MULTI_INSTANCE)
SetFilterMtMode("AVISource", MT_SERIALIZED)

#########################################
# External filters (guessed, might not be very stable)
##################################
#SetFilterMTMode("DitherPost", MT_SERIALIZED)
SetFilterMTMode("DitherPost", MT_NICE_FILTER)
SetFilterMTMode("Dither_Convert_8_to_16", MT_MULTI_INSTANCE)

# resize16 + crop16 are working even MT_NICE_FILTER in this sample
# when commenting out MDegrain2

#SetFilterMTMode("Dither_Resize16", MT_SERIALIZED) # when MDegrain2 used
SetFilterMTMode("Dither_Resize16", MT_MULTI_INSTANCE) # Freeze when MDegrain2 in clip

SetFilterMTMode("Dither_Crop16", MT_MULTI_INSTANCE)
SetFilterMTMode("Dither_quantize", MT_NICE_FILTER)
SetFilterMTMode("Dither_Out", MT_NICE_FILTER)
SetFilterMTMode("MDegrain2", MT_MULTI_INSTANCE)

SetFilterMTMode("MAnalyse", MT_MULTI_INSTANCE)
SetFilterMTMode("MRecalculate", MT_MULTI_INSTANCE)
SetFilterMTMode("MSuper", MT_MULTI_INSTANCE)

# Either Dither_Out or Dither_Resize16 should be MT_SERIALIZED when MDeGrain2 used
# otherwise avisynth+ hangs silently

#Load plugins explicitly
#-----------------------
DLLPath="..\8mmDLLs\"
#Loadplugin(DLLPath+"mt_masktools-26.dll") #Version 2.0.48.0 2012.04.02 needed for Dither_ package Access Viola on AVS+
Loadplugin(DLLPath+"mt_masktools-25.dll") #Version 2.0.48.0 2012.04.02 needed for Dither_ package
Loadplugin(DLLPath+"dither.dll") #Version ????? 2013.10.26
Import(DLLPath+"dither.avsi")
LoadPlugin(DLLPath + "mvtools2.dll") # Version 2.6.0.5 MSuper, MDegrain2, etc.. (2012.07.17 Dither 1.24 package modified version)
LoadPlugin(DLLPath+ "Rgtools.dll") # tp7 alternative new package for Clense, RemoveGrain

#keep it the last line please
if (FunctionExists("avstp_set_threads")) {
avstp_set_threads(0, 1) #disable threading in dither
}

#################################
# load video
#################################
Avisource(film).killaudio().assumefps(play_speed).trim(trim_begin,trim_end).converttoYV12()
clp=last

#--------------------
#-- Start of MDegrain part.
# variables for denoising (MDegrain)
denoising_strength= 600 #denoising level of first denoiser: MDegrain()
block_size= 8 #block size of MVDegrain
block_over= 4 #block overlapping

prefiltered = Removegrain(clp,mode=2)
superfilt = MSuper(prefiltered, hpad=32, vpad=32,pel=2)
super= MSuper(clp, hpad=32, vpad=32,pel=2)

halfblksize= (block_size>4) ? block_size/2 : 4
halfoverlap= (block_over>2) ? block_over/2 : 2

bvec1 = MAnalyse(superfilt, isb = true, delta = 1, blksize=block_size, overlap=block_over,dct=0,chroma= _grey ? false : true)
bvec1 = MRecalculate(super, bvec1, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

fvec1 = MAnalyse(super, isb = false, delta = 1, blksize=block_size, overlap=block_over,dct=0,chroma= _grey ? false : true)
fvec1 = MRecalculate(super, fvec1, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

bvec2 = MAnalyse(super, isb = true, delta = 2, blksize=block_size, overlap=block_over,dct=0,chroma= _grey ? false : true)
bvec2 = MRecalculate(super, bvec2, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

fvec2 = MAnalyse(super, isb = false, delta = 2, blksize=block_size, overlap=block_over,dct=0,chroma= _grey ? false : true)
fvec2 = MRecalculate(super, fvec2, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

#MDegrain2 from the modified Dither package version mvtools2
clp.MDegrain2(super, bvec1,fvec1,bvec2,fvec2,thSAD=denoising_strength)
GreyScale()
#-- End of MDegrain part
#---------------------------

Dither_Convert_8_to_16()
Dither_Crop16(16,16,-16,-16) # just to have a crop and resize
Dither_Resize16(720,576) # back to original for the sake of demo
DitherPost()

#for AVS+ enable MT
Prefetch(8)

[Zep]

No prefetch is technically Prefetch(0). Prefetch(1) is currently equivalent, but it is not guaranteed.

Prefetch itself predicts what frame will be requested next and fetch that frame in advance (in available thread). Prefetch(0) means to not use prefetching. Prefetch(1) will create a thread pool of one thread and do prefetching solely on that thread. Thus there is no guarantee that Prefetch(1) will request frame from its upstream filter in same order as it is requested (but it is currently this way)

mode 1 working then seems a bit odd in Prefetch(1) with lansings filter choice. Was mode 1 really being utilized?

[TurboPascal7]

Note to all people having any kind of issues with mt build: first of all, try removing SetFilterMTMode("", MT_MULTI_INSTANCE) from your script. I have absolutely no idea why this was introduced in the first place since it only creates problems. Yes, this will cripple performance but no one promised you it'll be easy.

Second - setting any MT mode on script functions does nothing. For example, SetFilterMTMode("Dither_Crop16", MT_MULTI_INSTANCE) doesn't change anything. You can put any other MT mode there and it will still work they way it works by default. Only plugin functions matter.

Third - situation where one plugin works with mode 1 when used before filter X and only with mode 2 when X is not present is pretty much impossible (assuming there are no bugs in the core of course :)). MT mode is a property of a filter and it doesn't depend on where it's used, only on filter implementation and input parameters (unless the filter is completely broken). I understand this is not helping much and thank you for posting scripts with this kind of problems, but most likely it means that MT mode of one of the filters was not "guessed" right. Hopefully we can get authors of some plugins who are still alive (dither for instance) to supply correct MT modes for every function in their plugins, either as a registration routine in the plugin itself, or SetFilterMtMode line. If we can't, well... we'll be guessing.

EDIT: as innocenat pointed out on IRC, it's actually possible to have filters changing their behavior if they're called before a filter with MT mode 3. All filters are getting serialized and any MT mode you're applying to them should not matter. This is an implementation detail though and you should not rely on it in any way since it might get changed without notice.

Now, to Prefetch(1)/Prefetch(0) issue. As ultim explained (no one bothered to read the full MT implementation tbh because it feels a bit (over)complicated), the number that you supply to prefetch is the number of prefetching threads. So in theory Prefetch(1) should give you two threads - one main thread and one prefetching thread. Prefetch(0) should disable any multithreading.

As usual, practice doesn't match theory very well and Prefetch(1) actually gives you one working thread (dunno if it's the main or the created one). Yes, this is a bug and it most likely will be fixed.

But if you think about it, this API design is very bad. Most users don't know and don't need to know the difference between prefetching threads and the main thread. Adding Prefetch(4) to their script, they absolutely reasonably expect it to work in 4 threads, trying to explain to them that it's actually 5 would be silly. So this bug kinda makes the API better.

In short - yes, currently Prefetch(1) and Prefetch(0) are almost equivalent. In the future one of these might error out, or this behavior will be kept (probably more reasonable).

[pinterf]

TurboPascal7, thank you the script function vs mt mode note, although I knew it, did not check which Dither function was implemented in script.

Luckily I have cought an error message displayed for a few tenth of seconds (then made a screenshot video)
"Dither_resize16: resizing ratio too low or kernel support too high." In MT_SERIALIZED mode it does not occur. (v1.24)

[mark0077]

Guys maybe this is a stupid question, but has someone (or is it even possible) to use this with SVP? I have been trying with no luck to replace my SetMTMode calls with SetFilterMTMode calls towards SVP functions. Maybe its not even possible unless SVP add support in their dlls?

[hank315]

Just installed Avisynth+ to run some tests as frame server for HCenc.
Most tests finished OK but the most basic test failed: Avisource input only.
Frames seem to have a wrong pitch.

Source is RGB in an AVI container.
Probably I'm doing something stupid because if this is a bug it should have been reported earlier I guess.

Used script:
s = AVISource("part2.avi")
s = ConvertToYV12(s)
return(s)

Source can be found here (http://hank315.nl/files/part2.7z).


BTW, Avisynth 2.6.a5 and 2.5.8 handled the source OK.
Is it correct frame->GetPitch is mod32 in Avisynth+ ?

[TurboPascal7]

Is it correct frame->GetPitch is mod32 in Avisynth+ ?
Most of the times - yes. But you still can get unaligned input after crop(align=false) for example.

[Zathor]

ClearAutoloadDirs()
AddAutoloadDir("C:\Program Files\MeGUI\tools\avs\plugins")
LoadPlugin("C:\Program Files\MeGUI\tools\avisynth_plugin\NicAudio.dll")
RaWavSource("E:\audio.wav")
SSRC((AudioRate()*25)/24).AssumeSampleRate(AudioRate()).TimeStretch(pitch=Float((AudioRate()*25)/24)*100.0/Float(AudioRate()))
return last
"An error occurred: Input audio sample format to SSRC must be float."

Seems portable version is AviSynth+ (ClearAutoloadDirs(), AddAutoloadDir()), and I think have a new version of SSRC but maybe without the automatic conversion of input data type than AviSynth 2.5.8 have.
When source was decoded from AC3, DTS, etc. with NicAudio or BassAudio the audio samples are float, but from losless sources the audio samples are int, and SSRC from AviSynth+ show the Error.

It seems that the SSRC filter in AviSynth+ cannot accept int audio while the AviSynth filter does. Reason seems to be that the automatic conversion mentioned here (http://avisynth.nl/index.php/ConvertAudio) seems to be not working. Is that a missing feature/intended or a bug?

[Zep]

But if you think about it, this API design is very bad. Most users don't know and don't need to know the difference between prefetching threads and the main thread. Adding Prefetch(4) to their script, they absolutely reasonably expect it to work in 4 threads, trying to explain to them that it's actually 5 would be silly. So this bug kinda makes the API better.

In short - yes, currently Prefetch(1) and Prefetch(0) are almost equivalent. In the future one of these might error out, or this behavior will be kept (probably more reasonable).

Thanks turboPascal your post was exactly the info i was looking for from a high level users eyes. For me I don't need the super low level details but I do want to know enough so i can tweak performance or at least a good guess as where to start/try.

thanks

[qyot27]

It seems that the SSRC filter in AviSynth+ cannot accept int audio while the AviSynth filter does. Reason seems to be that the automatic conversion mentioned here (http://avisynth.nl/index.php/ConvertAudio) seems to be not working. Is that a missing feature/intended or a bug?
It's by design:
https://github.com/AviSynth/AviSynthPlus/commit/d40f4a2e833998199dec0e0e4737cf69c4607b6c


There was also a forum post that mentioned it or explained the rationale (that it shouldn't do that kind of conversion without the user's knowledge), but I can't find it right now.

[Zathor]

Thanks, I have added ConvertAudioToFloat() to the script logic.

[zerowalker]

Here is a Bug Report, a Script that fails for no reason while it works on AVS 2.6 according to another user.

http://forum.doom9.org/showthread.php?p=1670117#post1670117

Hope that helps.

[innocenat]

Just stating in this topic also: the above issues happens in r1561 but not r1576. No fix needed. Refer to linked topic for more information.

[TurboPascal7]

Some testing done this afternoon with 1080P material.
The following hangs in MeGui preview:
LoadPlugin("D:\eseguibili\media\DGDecNV\DGDecodeNV.dll")
SetFilterMTMode("", 2)
SetFilterMTMode("DGSource", 3)
DGSource("E:\in\something.dgi")
#CompTest(1)
ChangeFPS(last,last,true)
SMDegrain (tr=6,PreFilter=2,thSAD=600,contrasharp=false,refinemotion=true,lsb_out=true,plane=0,chroma=false,Globals=2)
SMDegrain (tr=6, thSAD=600,contrasharp=false,refinemotion=true,lsb_in =true,plane=0,chroma=false,Globals=1,mode=6)
Prefetch(3)
Both works ok under AviSynth_MT from Set, which is even a bit faster.
I can confirm it being very slow (on 1080p) and probably unstable but it doesn't hang out completely for me in avspmod. Please try to remove SetFilterMTMode("", 2) line and see if it works (performance aside). This is most likely caused either by some globals magic or wrong mt mode specified for some filters.

mpeg2source doesn't seem to work with avs+ mt on. I have a 1080p mpeg2 video which I indexed with dgindex, and trying to test with mt on, it gave me a green screen onload with avspmod, and when I try seek, the program just hangs. All three mt mode will hang.


SetFilterMTMode("MPEG2Source", 2)
MPEG2Source("sample.d2v", cpu=0)
Prefetch(4)

Can't reproduce. Mode 1 is indeed broken but that's to be expected. Modes 2 and 3 work without issues here. Does it happen with some specific source or any sources in general?

Most tests finished OK but the most basic test failed: Avisource input only.
Frames seem to have a wrong pitch.
Source is RGB in an AVI container.
Used script:
s = AVISource("part2.avi")
s = ConvertToYV12(s)
return(s)

Source can be found here (http://hank315.nl/files/part2.7z).

Reproduced, reposted (https://github.com/AviSynth/AviSynthPlus/issues/36)on github. Avisource indeed can't handle mod32 pitch in this case. I will look into later this week.

pinterf
Firesledge added functions from dither here (https://pad.riseup.net/p/avs_plus_mt_modes) and they all seem to be NICE_FILTERs, so I guess the issue is somewhere in mvtools2. I'll probably play with it some time soon but wouldn't mind anyone testing it before me.

[ultim]

But if you think about it, this API design is very bad. Most users don't know and don't need to know the difference between prefetching threads and the main thread. Adding Prefetch(4) to their script, they absolutely reasonably expect it to work in 4 threads, trying to explain to them that it's actually 5 would be silly. So this bug kinda makes the API better.

I kindly disagree and will keep it this way even in the future. Hopefuly I will also find why Prefetch(1) is the same as Prefetch(0) and fix it.

Why I think it is better this way? First, the name of "Prefetch" is well chosen, because it describes exactly what the filter does: it prefetches frames. When inserted into a script, the user is basically telling the core a command to "Prefetch me frames in addition to the one you are working on now!". So the name is fine. So when you see the function declared as "Prefetch(nThreads)", what is the most intuitive way to interpret it? Isn't the first thing you think of "prefetch using nThreads threads"? I think it is, and this is exactly what AviSynth+ is doing now. Going TurboPascal's way would mean "Prefetch frames for me using one less threads than what I told you"! That is way more counterintuitive than the current implementation.

I do agree with TurboPascal that users shouldn't need to concern themselves with implementation details. They shouldn't, which is why I find it odd at all that he explained this. A higher number gives more threads: that is IMHO no less and no more than what a user must know in this case to find the optimal parameter for his number of cores, for best performance on his computer.

As for bugs, please remember, there is a reason this was published as a test build.

[ultim]

Well, bad news folks, we didn't get accepted for GSoC this year. Of course our "staff" will remain available for anybody wishing to pick up one of the projects, for help. There is always a next year, and we continue to hack around AviSynth ourselves too.

The preparation for GSoC was not all in vain though. We discussed a lot of things about the future of AviSynth's ecosystem. Thanks to GSoC, we managed to define what we want to do, defined our priorities and wishes clearly. As "by-products", we even got ourself a nice set of coding guidelines for future contributions, and improved our infrastructure just a bit further. Also, now that GSoC isn't playing for us in 2014, I can extend our ideas list with all those that we did discuss and want to have, but for various reasons deemed unfit for Google.

[TurboPascal7]

I kindly disagree and will keep it this way even in the future.
We can go full democracy and ask users to vote. Question is: "how many threads do you expect the following script to have? Assuming no filters are multithreaded internally."
DgSource("commie.dgi")
tfm().tdecimate()
debilinear(1280, 720)
gradfun3()
Prefetch(4)
My answer is 4. It's perfectly reasonable that you like your implementation (you wouldn't write it this way in the first place otherwise), but I believe it's non-obvious for anyone without programming experience.

[MasterNobody]

I don't see anything bad with having 5 threads when using Prefetch(4) if all hard work is done by this prefetch threads and one more (main) light thread is only needed to orchestrate other threads. That is mostly semantic of what x264 have for --threads and the only exception is --threads 1 when main thread and working threads become the one and the same thread.

[TurboPascal7]

I don't see anything bad with having 5 threads when using Prefetch(4) if all hard work is done by this prefetch threads and one more (main) light thread is only needed to orchestrate other threads. That is mostly semantic of what x264 have for --threads and the only exception is --threads 1 when main thread and working threads become the one and the same thread.
There's nothing wrong with having five or even more threads if all but four of them are idling most of the time, that's a minor implementation detail. But I thought the plan was to have main thread doing the same kind of stuff prefetch threads are doing (e.g. main thread requests frame 0, prefetch threads request frames 1, 2, 3 and 4).

[lansing]

Can't reproduce. Mode 1 is indeed broken but that's to be expected. Modes 2 and 3 work without issues here. Does it happen with some specific source or any sources in general?


mode 2 and 3 were fine with 480p source, but not for 1080 mpeg2 videos, I tried on a few clips already. This time only prefetch(0) work, all other prefetch would gave a green screen, and then after a few jumps in timeline, the program would hang.

[TurboPascal7]

mode 2 and 3 were fine with 480p source, but not for 1080 mpeg2 videos, I tried on a few clips already. This time only prefetch(0) work, all other prefetch would gave a green screen, and then after a few jumps in timeline, the program would hang.
Reproduced, thanks for the report.

[lansing]

Another problem, I tired running mctd script on a 1080 avc stream. I loaded the script into virtualdub and ran analysis pass, and it returned an error "could not allocate video frame. Out of memory".


dgsource("test.mkv", engine=1)
mctd()

[cretindesalpes]

I had this problem several times recently (using MT r1689) with scripts working perfectly with the vanilla Avisynth but failing to allocate frames in Avs+. Increasing the SetMemoryMax value is generally enough to make them work, but it also increases the overall memory usage, and you might end up closer to the fatal 4 GB barrier for a 32-bit process.

Maybe that releasing the cached frames from filters not used for a long time (for example after N GetFrame calls) would help? At least in my case there was important conditional filtering involved (lots of filters used once in a while) or long FFVideoSource+FFVideoSource+... chains.

[pinterf]

pinterf
Firesledge added functions from dither here (https://pad.riseup.net/p/avs_plus_mt_modes) and they all seem to be NICE_FILTERs, so I guess the issue is somewhere in mvtools2. I'll probably play with it some time soon but wouldn't mind anyone testing it before me.

1.) It says 1.25 version tested, typo? I know only Dither 1.24.

2.) Replaced mvtools2.dll from 2.6.0.5 of Dither 1.24 package => version 2.5.11.2 2011.04.18.
Now AVS+ works flawlessly.

Somehow mvtools2 2.6.0.5 causes Dither_Resize16 hang and give "Dither_resize16: resizing ratio too low or kernel support too high" message

3.) interesting (for me)
Setting MSuper to MT_NICE_FILTER instead of MT_MULTI_INSTANCE caused the first %Prefetch% frames currupted: they are dark/still noisy/ghosty


# some globals for the clip
play_speed=16
trim_begin=0 #120
seconds=120
trim_end=trim_begin + play_speed*seconds
_grey=false
film="c:\Tape13\Videos\1975_KeskenyFilm.avi"
#720x576 8mm film transfer

# AVS+ version
#SetFilterMTMode("", MT_MULTI_INSTANCE) #NOOO! Bad idea.

# SetFilterMTMode("Dither_Resize16", MT_SERIALIZED)
# MT_MULTI_INSTANCE/MT_NICE_FILTER Hangs with mvtools2 from 2.6.0.5 Dither package 1.24 )
# Error: "Dither_resize16: resizing ratio too low or kernel support too high"

SetFilterMTMode("Dither_Resize16", MT_NICE_FILTER) # with mvtools 2.5.11.2 NICE_FILTER_OK

SetFilterMTMode("DitherPost", MT_NICE_FILTER)
SetFilterMTMode("MDegrain2", MT_MULTI_INSTANCE)
SetFilterMTMode("MAnalyse", MT_MULTI_INSTANCE)
SetFilterMTMode("MRecalculate", MT_MULTI_INSTANCE)
SetFilterMTMode("MSuper", MT_MULTI_INSTANCE) #when NICE filter, the first %prefetch% frames bad: noisy/ghosty/dark

#Load plugins explicitly
#-----------------------
DLLPath="..\8mmDLLs\"
Loadplugin(DLLPath+"mt_masktools-25.dll") #Version 2.0.48.0 2012.04.02 needed for Dither_ package
Loadplugin(DLLPath+"dither.dll") #Version 1.24 2013.10.26
Import(DLLPath+"dither.avsi")
LoadPlugin(DLLPath + "mvtools2.dll") # Version 2.6.0.5 2012.07.17 from Dither 1.24: MT hangs Dither_Resize16 on AVS+
# version 2.5.11.2 works! 2011.04.18
LoadPlugin(DLLPath+ "Rgtools.dll") # tp7 alternative new package for Clense, RemoveGrain

#keep it the last line please
if (FunctionExists("avstp_set_threads")) {
avstp_set_threads(0, 1) #disable threading in dither
}

#################################
# load video
#################################
Avisource(film).killaudio().assumefps(play_speed).trim(trim_begin,trim_end).converttoYV12()
clp=last

#--------------------
#-- Start of MDegrain part.
# variables for denoising (MDegrain)
denoising_strength= 600 #denoising level of first denoiser: MDegrain()
block_size= 8 #block size of MVDegrain
block_over= 4 #block overlapping

prefiltered = Removegrain(clp,mode=2)
superfilt = MSuper(prefiltered, hpad=32, vpad=32,pel=2)
super= MSuper(clp, hpad=32, vpad=32,pel=2)

halfblksize= (block_size>4) ? block_size/2 : 4
halfoverlap= (block_over>2) ? block_over/2 : 2

bvec1 = MAnalyse(superfilt, isb = true, delta = 1, blksize=block_size, overlap=block_over,dct=0,chroma= _grey ? false : true)
bvec1 = MRecalculate(super, bvec1, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

fvec1 = MAnalyse(super, isb = false, delta = 1, blksize=block_size, overlap=block_over,dct=0,chroma= _grey ? false : true)
fvec1 = MRecalculate(super, fvec1, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

bvec2 = MAnalyse(super, isb = true, delta = 2, blksize=block_size, overlap=block_over,dct=0,chroma= _grey ? false : true)
bvec2 = MRecalculate(super, bvec2, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

fvec2 = MAnalyse(super, isb = false, delta = 2, blksize=block_size, overlap=block_over,dct=0,chroma= _grey ? false : true)
fvec2 = MRecalculate(super, fvec2, blksize=halfblksize, overlap=halfoverlap,thSAD=100)

#mvtools2 2.6.0.5 or 2.5.11.2
clp.MDegrain2(super, bvec1,fvec1,bvec2,fvec2,thSAD=denoising_strength)
GreyScale()
#-- End of MDegrain part
#---------------------------

Dither_Convert_8_to_16()
Dither_Crop16(16,16,-16,-16) # just to have a crop and resize
Dither_Resize16(720,576) # back to original for the sake of demo
DitherPost()

#for AVS+ enable MT
Prefetch(8)

[lansing]

I have some confusions with the mt mode. Let say if I have two filters, filter A and filter B. Filter A runs optimum at 5 threads, and filter B runs optimum at 7 threads. Then how many threads do I call at the end of the script? If I have prefetch(7) then filter A will be wasted, but if I have prefetch(5), filter B will be slowed down?

[cretindesalpes]

pinterf:

I tried your script and I can confirm it crashes or hang with dither.dll from Dither 1.24.0.
However it works with the new 1.25.0. I will release it soon.

[lansing]

I ran into a access violation error with this simple script


AVISource("sample.avi")
stab()
a = last
b = fft3dgpu(sigma=5)

mt_merge(a, b.mt_edge())


When I try to jump to another frame in timeline, a pop up error say "error requesting frame N, WindowsError: exception: access violation reading 0x00000002". When I replace fft3dGPU with fft3dfilter, the error go away.

[bxyhxyh]

Is there any list or Readme about avs+ functions?
I don't know any avs+ functions.

[ultim]

Just installed Avisynth+ to run some tests as frame server for HCenc.
Most tests finished OK but the most basic test failed: Avisource input only.
Frames seem to have a wrong pitch.

Source is RGB in an AVI container.
Probably I'm doing something stupid because if this is a bug it should have been reported earlier I guess.

Used script:
s = AVISource("part2.avi")
s = ConvertToYV12(s)
return(s)

Source can be found here (http://hank315.nl/files/part2.7z).


BTW, Avisynth 2.6.a5 and 2.5.8 handled the source OK.
Is it correct frame->GetPitch is mod32 in Avisynth+ ?

I wish everybody who reports a bug would attach both a small clip and script like you. It just makes things so much easier. Thanks for the report. Looking into the probem at this instant.

[ultim]

Is there any list or Readme about avs+ functions?
I don't know any avs+ functions.

Not yet. After ironing out the MT stuff, I will work on updating the docs.

[mastrboy]

Any chance Avisynth+ could add a file extension bool parameter to the function ScriptFile() ?
Example to show use case:

episode_name = ScriptFile(show_extension=false)
FFVideoSource(episode_name+".avi")
TFM(output=episode_name+"-tfm.log")

[TurboPascal7]

Any chance Avisynth+ could add a file extension bool parameter to the function ScriptFile() ?
Example to show use case:

episode_name = ScriptFile(show_extension=false)
FFVideoSource(episode_name+".avi")
TFM(output=episode_name+"-tfm.log")

Can't you just write an overload with that parameter and remove the extension using existing string functions? What's the point?

[mastrboy]

If I had that knowledge I wouldn't have requested the feature ;)

Could you point me in the right direction for doing so?

[TurboPascal7]

If I had that knowledge I wouldn't have requested the feature ;)

Could you point me in the right direction for doing so?

function scriptfile(bool show_extension) {
actual = scriptfile()
ext_position = actual.revstr().findstr(".")
without_ext = actual.leftstr(actual.strlen() - ext_position)
return show_extension ? actual : without_ext
}
The only downside is that the parameter is not named because you still need a way to call internal avisynth function inside this wrapper. I'm not sure how you can work around that, other than changing the name of the wrapper function.

EDIT: here's a "workaround".
function scriptfile(bool "show_extension") {
show_extension = default(show_extension, true)
path = scriptname()
with_ext = path.RightStr(path.strlen() - scriptdir().strlen() - 1)
ext_position = with_ext.revstr().findstr(".")
without_ext = with_ext.leftstr(with_ext.strlen() - ext_position)
return show_extension ? with_ext : without_ext
}

[mastrboy]

That was fast.

Working perfectly also, I just renamed the function to "_ScriptFile".

Thank you :)

[ultim]

mpeg2source doesn't seem to work with avs+ mt on. I have a 1080p mpeg2 video which I indexed with dgindex, and trying to test with mt on, it gave me a green screen onload with avspmod, and when I try seek, the program just hangs. All three mt mode will hang.


SetFilterMTMode("MPEG2Source", 2)
MPEG2Source("sample.d2v", cpu=0)
Prefetch(4)


Please provide a sample clip.

[lansing]

I had this problem several times recently (using MT r1689) with scripts working perfectly with the vanilla Avisynth but failing to allocate frames in Avs+. Increasing the SetMemoryMax value is generally enough to make them work, but it also increases the overall memory usage, and you might end up closer to the fatal 4 GB barrier for a 32-bit process.


It's getting so frustrated for me with this issue, to a point that it's just literally unusable. When I try to encode a two hour long 480p video, after 30% in the process, it's telling me not enough memory. And when I try to mount the script through avfs to load in premiere, after 20 seeks in timeline, the program will exit by itself because of low in memory. I set setmemorymax to 3000 and it has no effect.

[innocenat]

There is a memory leak problem that just get fixed in Overlay filter in all MT built prior to last Monday. If you script doesnt use overlay can you post sample script?

[lansing]

There is a memory leak problem that just get fixed in Overlay filter in all MT built prior to last Monday. If you script doesnt use overlay can you post sample script?

the latest avs+ version I have is r1689. All my scripts contain qtgmc.

[ultim]

the latest avs+ version I have is r1689. All my scripts contain qtgmc.

It also has been discovered that in some cases the MT protection is not correctly applied to all filters. This is also the source of multiple problems reported here on the forums, like the issue with DSS+AudioDub, or with Mpeg2Source(1080i). Theoretically it can also cause leaks in filters.

A fix for this is on the way, but it may not be the cause for you at all. To make sure your leak gets fixed in the next public build, please post your script, and the format+resolution of your video (or even better, a short sample clip).

[lansing]

It also has been discovered that in some cases the MT protection is not correctly applied to all filters. This is also the source of multiple problems reported here on the forums, like the issue with DSS+AudioDub, or with Mpeg2Source(1080i). Theoretically it can also cause leaks in filters.

A fix for this is on the way, but it may not be the cause for you at all. To make sure your leak gets fixed in the next public build, please post your script, and the format+resolution of your video (or even better, a short sample clip).

This is the script I used with mounting. I don't have MT on. The video format is mpg, and resolution is 720x570.


qtgmc(preset="fast")

deblock_qed(quant1=50, quant2=26)
Crop(0, 0, -0, -6)


I loaded the video into premiere and opened task manager to keep track of the memory usage. Every time I seek, the memory usage of both premiere and Pismo file mount audit went up. In about 20 seeks, the mounting software would went over 3G and premiere over 4G, and eventually ran out of memory. I tested the same script with SEt's 2.6mt, the prior never go over 600MB and premiere never go over 2GB.

[Evil_Burrito]

woah woah woah, lansing, are you running that script into premier? Frameserving to an full editing application is supposed to only include source filters. Yes you "can" do it, but it is highly unrecommended (especially with heavy filters like qtgmc).

[lansing]

woah woah woah, lansing, are you running that script into premier? Frameserving to an full editing application is supposed to only include source filters. Yes you "can" do it, but it is highly unrecommended (especially with heavy filters like qtgmc).

my script wasn't "that" heavy compare to mctd, and I'm using preset=fast, and my resolution wasn't 1080. And I ran it with no problem using avs 2.6mt.

[ultim]

woah woah woah, lansing, are you running that script into premier? Frameserving to an full editing application is supposed to only include source filters. Yes you "can" do it, but it is highly unrecommended (especially with heavy filters like qtgmc).

I see no reason why frameserving to a "full editing application" like Premier shouldn't work reliably. So I still consider lansing's report a valid bug report and I will look into it. Right now I suspect it is not a leak, but some weird cache-behaviour which should be fixed, but this is just a guess right now.

[Boulder]

I'm going to hop on from Avisynth MT (SEt's latest v2.6 build) to Avisynth+ just for some testing. Are there going to be any expected difficulties that I definitely should be aware of or can I simply install it and start fiddling on things?

[ultim]

I'm going to hop on from Avisynth MT (SEt's latest v2.6 build) to Avisynth+ just for some testing. Are there going to be any expected difficulties that I definitely should be aware of or can I simply install it and start fiddling on things?

This bug (http://forum.doom9.org/showpost.php?p=1672010&postcount=678) makes it hard to tell different issues apart from each other. For this reason I would suggest you wait for the next public build, which will have not only this but also other important fixes in place, some already corrected in the repository.

[aegisofrime]

It also has been discovered that in some cases the MT protection is not correctly applied to all filters. This is also the source of multiple problems reported here on the forums, like the issue with DSS+AudioDub, or with Mpeg2Source(1080i). Theoretically it can also cause leaks in filters.

A fix for this is on the way, but it may not be the cause for you at all. To make sure your leak gets fixed in the next public build, please post your script, and the format+resolution of your video (or even better, a short sample clip).

Is this bug present in the last stable build, the one before MT was implemented? Because I have been finding that I have this script that always seem to crash in the same spot, and I guess it might be due to a leak.

Here's my script:


LoadPlugin("C:\DGDecIM\DGDecodeIM.dll")
DGsource("I:\Test\F1_T2_Video - .h264",engine=1)
AssumeTFF()
QTGMC(Preset="Very Slow")
Crop(0,0,0,-8)

I know that DGSourceIM is still in beta, but the script also crashes with the more mature DGAVCSource...

I would love to post my source, but it's a 20GB H264 from a Blu-Ray so that might be impractical...

[ultim]

Is this bug present in the last stable build, the one before MT was implemented?

No, it is not present in the stable build. That bug is strictly MT-related, and even in MT-builds only manifests if you have a Prefetch() call in your script.

I'll look at that bug of yours. It is also surely not due to the new caches, because the stable build still uses the old cache.

[Evil_Burrito]

I guess I stand semi-corrected. I did not mean to say it will be unstable, I just meant it will slow down your workflow while you are editing. Is this statement incorrect?

[real.finder]

hi :)

In order to avoid problems with fft3dgpu and Internal mt and other problems, it would be better to have mode 0, SetFilterMTMode("", none)

in mode 0 no mt mode will use, like works in regular avs

and for Prefetch, the 0 should be auto like regular avs mt, and make it default

thanks

[innocenat]

hi :)

In order to avoid problems with fft3dgpu and Internal mt and other problems, it would be better to have mode 0, SetFilterMTMode("", none)

in mode 0 no mt mode will use, like works in regular avs

and for Prefetch, the 0 should be auto like regular avs mt, and make it default

thanks

I am not sure what you mean. If you don't use Prefetch (or Prefetch(0)) everything will behave like regular non-multithreaded Avisynth.

If you are saying that there should be a mode where no MT mode is set for some filter in multithreaded filter chain, that would be impossible. The problem is that we are dealing with "filter chain" where its downstream filter can request frame from it however it wants. This is where MT mode come into play: it controls how the downstream filter request the frame from parent filter. Sadly, it still isn't perfect and we are still having problem.

[real.finder]

If you are saying that there should be a mode where no MT mode is set for some filter in multithreaded filter chain

yes, that what I mean, some thing like I said here (http://forum.doom9.org/showthread.php?p=1658255#post1658255) previously

that would be impossible

what about some thing like mp_pipeline but internal?

-----------


About my words for Prefetch, is meant for it's number of threads

[ultim]

hi :)

In order to avoid problems with fft3dgpu and Internal mt and other problems, it would be better to have mode 0, SetFilterMTMode("", none)

in mode 0 no mt mode will use, like works in regular avs

and for Prefetch, the 0 should be auto like regular avs mt, and make it default

thanks

I've had a similar idea. There would be 4th MT-mode, for example MT_DISABLED/MT_SINGLE_ONLY/MT_NO_MT (pick any naming you like), and if a filter uses this mode in a multi-threaded filter chain, the user would get an error that MT with that filter is not supported.

[ajp_anton]

I just realized that Overlay doesn't do its processing in RGB. Could this be "fixed"?

[Gavino]

I just realized that Overlay doesn't do its processing in RGB. Could this be "fixed"?
For RGB inputs, Layer() is preferable to Overlay() for most purposes. It's faster and uses less memory, but the range of features is not exactly the same, so it cannot do some things that Overlay() can.

[ajp_anton]

Why do we even have all those different functions? Overlay's "opacity" parameter could replace even Merge. Woudn't it be more practical to just have one function to do all of this?

[TurboPascal7]

Why do we even have all those different functions? Overlay's "opacity" parameter could replace even Merge. Woudn't it be more practical to just have one function to do all of this?
There are multiple ways of doing pretty much anything in avisynth, nothing really wrong with that.

I agree that the overlay/layer situation is a bit confusing, but without breaking compatibility with existing scripts, the best thing we could do is to unify the codebase and make one function a wrapper (possibly a script one) of another.

[ajp_anton]

Merge could also be implemented as a wrapper, unless you can make it faster by skipping all the extra stuff.
But yes, Overlay and Layer should be merged into one function with all the features and supported colorspaces of both, without doing unnecessary internal conversions.

[DeathAngelBR]

Well, hello.

I'm just wondering if a previous filter chain used for encoding with avisynth is going to work with avisynth+. I'm getting a bit more performance with the same settings in avs+, so it made me think that maybe the plugins/filters aren't working properly.

LoadPlugin("D:\Edição de video\MeGUI\tools\dgavcindex\DGAVCDecode.dll")
LoadPlugin("D:\Arquivos de programas\AviSynth 2.5\plugins\removegrain\RemoveGrainSSE2.dll")
LoadPlugin("D:\Arquivos de programas\AviSynth 2.5\plugins\removegrain\RepairSSE2.dll")
LoadPlugin("D:\Arquivos de programas\AviSynth 2.5\plugins\removegrain\RSharpenSSE2.dll")
LoadPlugin("D:\Arquivos de programas\AviSynth 2.5\plugins\outros\mt_masktools-26.dll")
LoadPlugin("D:\Arquivos de programas\AviSynth 2.5\plugins\outros\mvtools2.dll")
LoadPlugin("D:\Arquivos de programas\AviSynth 2.5\plugins\outros\AddGrainC.dll")
LoadPlugin("D:\Arquivos de programas\AviSynth 2.5\plugins\outros\dither.dll")
Import("D:\Arquivos de programas\AviSynth 2.5\plugins\outros\dither.avs")

video_ep04 = AVCSource("I:\BDMV Symphogear G Vol. 2\BDMV\STREAM\00004.dga")
video_ep05preview = AVCSource("I:\BDMV Symphogear G Vol. 2\BDMV\STREAM\00013.dga")

video_ep04 ++ video_ep05preview

addborders(0,4,0,4)
super = MSuper(pel=2, sharp=1)
backward_vec1 = MAnalyse(super, isb = true, delta = 1, overlap=4,mt=true)
forward_vec1 = MAnalyse(super, isb = false, delta = 1, overlap=4,mt=true)
MDegrain1(super, backward_vec1,forward_vec1,thSAD=64)
crop(0,4,0,-4,align=true)
dither_convert_8_to_16().dither_resize16(1280,720).smoothgrad().gradfun3(lsb_in=true,lsb=true,smode=2,mask=3).dither_out()

[ajp_anton]

Oh, and call me retarded, but where do I get revisions newer than r1576?
At least in that version, Layer completely ignores the overlay clip's alpha and assumes it's zero everywhere.
m = colorbars
c = blankclip(m,color=$808080)
o = blankclip(m,color=$ff0000)

o = mask(o,m)

layer(c,o)
#o.showalpha #to see that the mask is indeed there
Overlay red on top of grey with the colorbars greyscale as mask. Works in 2.6.

[CarlPig]

Oh, and call me retarded, but where do I get revisions newer than r1576?
At least in that version, Layer completely ignores the overlay clip's alpha and assumes it's zero everywhere.
m = colorbars
c = blankclip(m,color=$808080)
o = blankclip(m,color=$ff0000)

o = mask(o,m)

layer(c,o)
#o.showaplha #to see that the mask is indeed there
Overlay red on top of grey with the colorbars greyscale as mask. Works in 2.6.

I think this is the newest release: http://forum.doom9.org/showthread.php?p=1666364#post1666364

[ajp_anton]

Thanks.
Nope, Layer doesn't work there either. This is the reason I used Overlay in the first place, couldn't figure out how to get Layer to work so I switched. Turns out the fault wasn't mine =).

[qyot27]

I'm getting a bit more performance with the same settings in avs+, so it made me think that maybe the plugins/filters aren't working properly.
What? This is right there in the list of features for avsplus:
Changes compared to AviSynth

64-bit version available
Faster script startup and substantially faster core filters
Support for multiple ("shadow") plugin directories
Autoloading of C-plugins
Script language extensions, with support for multiline conditionals and loops.
Improved still image support and TimeStretch
Justifiably, anything that depends on those core filters would get a boost too. To say nothing of the cache changes if you're using r1689 or higher.


Not to mention that a few of those filters have rewritten versions with avsplus in mind, and you'd get better (and probably more accurate, or at least just as accurate) results from using FFMS2 or LSMASHSource as your source filter, at the very least because those can use current builds of libavcodec, and libavcodec has most certainly gotten faster at decoding H.264 over the last 5 years since the last public release of DGAVCDec.

[TurboPascal7]

Thanks.
Nope, Layer doesn't work there either. This is the reason I used Overlay in the first place, couldn't figure out how to get Layer to work so I switched. Turns out the fault wasn't mine =).
Is it avs+-specific bug or do you experience the same problem with regular avs 2.6?

[ajp_anton]

avs+ specific. 2.6 works perfectly as soon as I replace avisynth.dll.

[TurboPascal7]

avs+ specific. 2.6 works perfectly as soon as I replace avisynth.dll.
Then please do provide an example script.

[ajp_anton]

Already did:m = colorbars
c = blankclip(m,color=$808080)
o = blankclip(m,color=$ff0000)

o = mask(o,m)

layer(c,o)
#o.showalpha #to see that the mask is indeed thereColorbars just to create some non-uniform mask, and use it to overlay uniform red into uniform grey.

[TurboPascal7]

Fixed (https://github.com/AviSynth/AviSynthPlus/pull/42). Thanks for the report.

[ajp_anton]

Is there any reason for r1689 to use more memory than r1576? (single-threaded)
This might be crazy, but I use Avisynth to generate a very large image full of data (both text and some simple graphics) that needs updating every day. r1689 crashes when I make it too large, r1576 doesn't.
Script is very long and includes some sensitive information. I could modify it slightly and send via PM if needed.

[vcmohan]

At the outset I request you all to make allowance to my limited knowledge of C++ programming.
I have read a number of posts starting from http://forum.doom9.org/showthread.php?p=1651085#post1651085 on multithreading and plugin compatibilities. I could not get a clear view of whether a plugin developer should in someway inform Avisynth+ whether it is MT compatible (if so in what way), or 'user' need to inform in his script. At someplace it was also mentioned that C++ interface is inferior to C interface.

I also read that for the present plugin developers may stick to Avisynth 2.6 plugin development, as Avisynth+ interface is likely to change.

I appreciate that this work is in progress and so is avisynth 2.6 alpha. However instead of compatibility for all versions, plugin developers (who are interested) may be guided as to which features they need to have to be compatible to the new avisynth+ or 2.6.

I will be grateful if all such instructions be mentioned and periodically updated either in a seperate thread or at the beginning of this thread.

[innocenat]

At the outset I request you all to make allowance to my limited knowledge of C++ programming.
I have read a number of posts starting from http://forum.doom9.org/showthread.php?p=1651085#post1651085 on multithreading and plugin compatibilities. I could not get a clear view of whether a plugin developer should in someway inform Avisynth+ whether it is MT compatible (if so in what way), or 'user' need to inform in his script. At someplace it was also mentioned that C++ interface is inferior to C interface.

I also read that for the present plugin developers may stick to Avisynth 2.6 plugin development, as Avisynth+ interface is likely to change.

I appreciate that this work is in progress and so is avisynth 2.6 alpha. However instead of compatibility for all versions, plugin developers (who are interested) may be guided as to which features they need to have to be compatible to the new avisynth+ or 2.6.

I will be grateful if all such instructions be mentioned and periodically updated either in a seperate thread or at the beginning of this thread.

The current MT version of Avisynth+ is still in development and is only for testing purposes. Thus, it is not (and should not) really be officially documented in the beginning of this thread.

As for the plugin, plugin author should inform the Avs+ core of its preferred operating mode (and the API is Avs2.6-compatible). However, while current API for this is are not likely to change, it is still not guaranteed to not be changed in final stable version of Avs+.

Avs+ fully support plugin written in 2.6 api (and if it isn't, it's a bug). For now it is recommended that you stick to original 2.6 API. There are currently extended API in the development version of avs+ core, but as I said, it is still not stable and should not be used by plugin as there are no compatibility guarantee between each testing revision. Furthermore, usage of those API will make your plugin not compatible with original Avs2.6.

Hope this helps.

[turbojet]

I noticed resizers were worked on 2 months ago. Are there any mt builds around that includes this?

[innocenat]

I noticed resizers were worked on 2 months ago. Are there any mt builds around that includes this?

You should be able to find one in https://mega.co.nz/#F!ZEtEELLB!DovfKg-0imp8R9JlasCzzQ Note that MT build is experimental.

[turbojet]

1689 has the fixes (https://github.com/AviSynth/AviSynthPlus/pull/27)? That's the same one Groucho and I tested that was about 2% slower with lanczos and spline.

The dll's version is 2.6.0.5 can that change to revision number in the future?

[innocenat]

1689 has the fixes (https://github.com/AviSynth/AviSynthPlus/pull/27)? That's the same one Groucho and I tested that was about 2% slower with lanczos and spline.

The dll's version is 2.6.0.5 can that change to revision number in the future?

Yes. r1689 should already have that fixes. Admittedly, I don't have any AMD to test on, so I don't know prior to your tests whether it is faster or slower on AMD's CPU. I do have a few more idea to optimize the resizer, but I haven't got around to do it yet.

[turbojet]

Ok, I was getting excited about something I already had then. The performance difference is about the same on Groucho2004's intel then it is on my amd. Bicubic is a lot faster maybe there's something in those that can improve lanczos, spline, bilinear, others haven't been tested.

[turbojet]

The riseup pad (https://pad.riseup.net/p/gfyPtMbihf6q) has been deleted. Is there another source for the mtmode avsi?

Might be nice to have it on the main site or original post of this thread.

[TurboPascal7]

I dunno where you got that link but it's wrong. Here's the right one (https://pad.riseup.net/p/avs_plus_mt_modes).

[turbojet]

Thanks, it was the from the first post after searching this thread for 'pad' but it was for installer translation. I should have paid more attention.
Some questions about mt:
1. Is an avsi the plan going forward with MT modes?
2. If so is there plans on having some sort of an update to a moderated avsi, or maybe included in new versions?
3. Is defining it inside the dll a lot of work?
4. Is there any thoughts on making mt opt-out instead of opt-in after it's stable? It seems to work fine so far for me, it often doesn't speed it up but it doesn't have any noticeable disadvantages either.

[cretindesalpes]

Hi,

This is probably related to this bug (http://forum.doom9.org/showthread.php?p=1669861#post1669861).

FFVideoSource ("bad-stack16.mkv", enable10bithack=true)
Video sample (12 MB) (http://www.mediafire.com/download/r4hfullc4tsy6da/bad-stack16.mkv)
Tested with Avisynth+ MT r1689 and FFMS2 r734-libav patched by SAPikachu.
Chroma planes have a wrong pitch. Happens only in stack16 mode. Everything works correctly in Avisynth 2.6 alpha 5.

[aegisofrime]

Hi, this thread has been pretty quiet lately. :( How's the progress on the MT version?

[Jeroi]

Does this project be active, github states that nothing happened in few months...? I would really like the features that this project aims expecially the directshow api support.

[TurboPascal7]

I myself lost interest in avisynth development (maybe temporarily) and all other people are a bit busy with other things right now. But ultim should be "back" some time soon, hopefully.
So no, right now the project obviously isn't "active".

[nickintheforest]

MT-enabled AviSynth+ triggers a latent bug in AvsPmod. Until a new version of AvsPmod is officially released, use this (http://www46.zippyshare.com/v/29821586/file.html) build. A thousand thanks to vdcrim for the fix.

Can anybody reupload that build?

Reel.Deel
thanks

[Reel.Deel]

@nickintheforest

Here's an active link:

AvsPmod-2.5.1-r426-x86-04874ed.7z (http://files.line0.in/builds/AvsPmod-2.5.1-r426-x86-04874ed.7z)

[real.finder]

Waiting for your return, especially ultim
I hope this calmness will be the calm before the storm

[zerowalker]

TurboPascal7,

If you find the interest, i would like to request ImageSource to be upgraded to support .Webp.

Probably alone in this, but might as well mention it if you happen to have any interest in it;P

[Jeroi]

Avisynth+ does not seem to be the future atm, not much development going, even Avisynth is getting more development these days...

[ryrynz]

That benefits Avisynth+ too.. hopefully a stable MT build later this year.

[aldix]

Dunno... using avisynth+ MT atm without serious issues...

[aegisofrime]

It seems like the alternative Avisynth developments are cursed... Anybody remember JoshyD's Avisynth 64 builds? Things were going great for a while, and he also ported many filters to 64-bit. Then he disappeared...

I hope the same isn't happening to ultim :scared:

[jpsdr]

Anybody remember JoshyD's Avisynth 64 builds?
Yes i remember.
But not all are cursed. Even if i don't use it, i think Set MT version is still alive, no ?

[StainlessS]

Is there a suggested alternative download link for A+, tried about 6 times to download, get an
"Invalid user name or password, or Access Denied" error message half way though.

EDIT: The installer.

[Groucho2004]

Is there a suggested alternative download link for A+, tried about 6 times to download, get an
"Invalid user name or password, or Access Denied" error message half way though.

EDIT: The installer.
Weird, works for me.

[ARDA]

Is there a suggested alternative download link for A+, tried about 6 times to download, get an
"Invalid user name or password, or Access Denied" error message half way though.

Just guessing, maybe you are logged in github

This is a direct link , it is working ok for me
https://github.com/AviSynth/AviSynthPlus/releases/download/Rel-r1576/AviSynthPlus-r1576.exe

if you don`t succeed, use this alternative of my own
https://mega.co.nz/#!REFXDTZb!g9LgEVXHSf92dvhZm5ksOIfDlcry8VPOwo2rbaH0mxA

[StainlessS]

Thank you Arda, I finally got it from 1st link (am not member on GitHub, so not signed in).
Thought I might finally give this a whirl, as the captain seems to have jumped ship, might be wise to unlash the life boats.

[Groucho2004]

Thought I might finally give this a whirl
There are some API differences in AVS+, one of which I stumbled upon a couple of days ago when I added the enumeration of plugin functions in the Avisynth Info Tool. The global variable "$PluginFunctions$" was removed because of the way AVS+ handles plugins. I believe you also use this variable in your RT_Stats code.

[StainlessS]

Goucho, I read somewhere that $PluginFunctions$ was being put back in, probably because of my use of it.

A bit later ... just checked my RT_Stats functions listing script, works just fine.

[Groucho2004]

just checked my RT_Stats functions listing script, works just fine.
Yes, but only after loading a script and initializing the plugin API.

Edit: Ha, I think I can trick it with a dummy call to "Invoke("LoadPLugin",...)". This way I don't have to load a script to get the external functions. :sly:

[StainlessS]

Ah,well, I only know it from the plugin side, if what I said makes sense to you and gave clue to a
workaround, that's great. Peace Bro.

[StainlessS]

Thank you Grouchy, I posted a script recently to give functions contained in an autoload only (from plugins dir) dll,
your Inkoke thing would probably (I think) also fix the issue I had. You are such a clever little lad :)
EDIT: This post: http://forum.doom9.org/showthread.php?p=1681523#post1681523

EDIT: I did try LoadPlugiin("PlugName.dll") but did not work, think you might need to know the name of a function/filter
and call it, to force it into existence (and of course if you knew names of functions, you would not be searching for them).

[zerowalker]

Is the MT working with smdegrain, if so is there any special settings/modes that needs to be used?

Thanks

EDIT:



SetFilterMTMode("FFT3Dgpu", 3)
SetFilterMTMode("", 2)

Avisource("Z:\Test.avi")
ConvertToYV12(matrix="Rec601", interlaced=false)

FFT3Dgpu(sigma=1.5, plane=3)

smdegrain(tr=3,lsb=true,thsad=300,lsb_out=true,search=3,refinemotion=true)
Dither_convey_yuv4xxp16_on_yvxx()
Prefetch(4)


I am guessing this should work from reading around, but not entirely sure.

[StainlessS]

Bump, dont miss last edit in my previous post Grouchy.

[Groucho2004]

I did try LoadPlugiin("PlugName.dll") but did not work, think you might need to know the name of a function/filter
and call it, to force it into existence (and of course if you knew names of functions, you would not be searching for them).
I'm doing this to force the invokation of "LoadPlugin":
try
{
AVSValue foo = AVS_env->Invoke("LoadPlugin", 0);
}
catch (AvisynthError e)
{
}
catch (...)
{
}



I'm not sure any more if we're talking about the same thing. You seem to be referring to scripting.

[Evil_Burrito]

Is the MT working with smdegrain, if so is there any special settings/modes that needs to be used? ...

SetFilterMTMode("FFT3Dgpu", 3)
SetFilterMTMode("", 2)
Avisource("Z:\Test.avi")
ConvertToYV12(matrix="Rec601", interlaced=false)
FFT3Dgpu(sigma=1.5, plane=3)
smdegrain(tr=3,lsb=true,thsad=300,lsb_out=true,search=3,refinemotion=true)
Dither_convey_yuv4xxp16_on_yvxx()
Prefetch(4)


I am guessing this should work from reading around, but not entirely sure.

Yes it will run, but mt scripts are not written the same way as using SEt's mt modes. Line 2 SetFilterMTMode("", 2) does not do anything because you didn't specify a filter inside "blank". Although, I myself do not 100% understand how to use the new avisynth+ mt, I do know that you have to specify the mode for each filter (excluding the core filters (and maybe a couple new filters specifically written for avisynth+?))
-----------------------
I have been meaning to ask a couple questions for months. Responders please don't lazy half answer my questions by just linking to an older post from this thread. I have been keeping up with this thread since it was started.

Could someone clarify the correct way to specify a filter when the *.dll and it's filters do not match. Which one of these is would be correct:
SetFilterMTMode("rgtools", 1)
SetFilterMTMode("RemoveGrain", 1) <-- This is what I have been using.
SetFilterMTMode("RemoveGrain", MT_NICE_FILTER)

Or do I even need to use SetFilterMTMode for this filter since it was made specifically for the new avisynth+?

(Masktools2.dll is another one, which I use often, that I am not exactly sure how to use multi threaded correctly)

Another question; SetMemoryMax(mb) is not used any more, right? I haven't added it to any scripts since I switched to avisynth+.

btw, I have been using both 32 and 64 bit versions of mt-r1689 (depending on the script) without any major problems. I just wanted to make sure I am writing mt scripts correctly. And I do know you can't call a script function .avsi in a SetFilterMTMode() line.

[TurboPascal7]

Line 2 SetFilterMTMode("", 2) does not do anything because you didn't specify a filter inside "blank".
Actually, it specifies MT mode for all filters which aren't specified explicitly.

Could someone clarify the correct way to specify a filter when the *.dll and it's filters do not match. Which one of these is would be correct:
SetFilterMTMode("rgtools", 1)
SetFilterMTMode("RemoveGrain", 1) <-- This is what I have been using.
SetFilterMTMode("RemoveGrain", MT_NICE_FILTER)

Or do I even need to use SetFilterMTMode for this filter since it was made specifically for the new avisynth+?

You don't have to specify anything with these (http://forum.doom9.org/showpost.php?p=1668101&postcount=92) builds. The correct usage is "RemoveGrain", not "rgtools". This has a problem of applying a wrong MT mode to the old kassandro's removegrain which might break like crazy. This problem has no real workaround now.

Whether you use 1 or MT_NICE_FILTER doesn't matter. MT_NICE_FILTER is just a variable that has the value of 1. You can set it to some other value in your script and break everything horribly.

Another question; SetMemoryMax(mb) is not used any more, right? I haven't added it to any scripts since I switched to avisynth+.
IIRC yes.

[zerowalker]

So smdegrain is compatible with SetFilterMTMode("", 2)?

Meaning my script will be able to run without an issue (no corrupted frames appearing and such)?
Cause i know from before that MT could cause random issues which wasn't something like "If it's working, it works", it could appear randomly.

[TurboPascal7]

So smdegrain is compatible with SetFilterMTMode("", 2)?

Meaning my script will be able to run without an issue (no corrupted frames appearing and such)?
Cause i know from before that MT could cause random issues which wasn't something like "If it's working, it works", it could appear randomly.
No. There is no guarantee that you'll get entirely correct output with any set of filters under any set of MT modes right how.

[zerowalker]

No. There is no guarantee that you'll get entirely correct output with any set of filters under any set of MT modes right how.

Okay that's sadly enough to make me not use it in this circumstance, even though the improvement in speed is quite significant.

Though, i don't mean that in a way to discourage the use or development of it.

Would be quite nice to have a working MT, even if optimization is needed for each plugin, it would be quite worth it for certain ones, QTGMC,smdregrain,MCTD, these scripts could really use any boost given, sadly they are very complex and use a lot of filters.

Must be frustrating work keeping check on those rather than individual plugins.


Anyhow, hope the work continues at any rate, anything is something at anytime;)

[Reel.Deel]

No. There is no guarantee that you'll get entirely correct output with any set of filters under any set of MT modes right how.

Yeah when I was testing Avisynth+ MT I came across a few plugins that would have corrupted output, mainly when using with MT_NICE_FILTER (mode 1). Fortunately most of these filters worked fine with MT_MULTI_INSTANCE (mode 2).

-----

To all who care:

Some of the plugins (I'm sure there's more) with corrupted output with MT_NICE_FILTER:


frfun7 (rev6) - 2013 version works fine
gradfun2db
tcanny
TCannyMod
TDeint
TEdgeMask
TEMmod

All of these plugins work correctly with MT_MULTI_INSTANCE. For additional read the comments here (https://pad.riseup.net/p/avs_plus_mt_modes).


Anyways, I hope ultim is doing good...awaiting his return to continue testing...

[Evil_Burrito]

TurboPascal7, thank you for clarifying.

[zerowalker]

I seem to get crashing when i run this script, at least after a long time (Like 7 hours or something).


Avisource("Z:\Testgraph MagicYUV - FFT3DGPU.avi")

ConvertToYV12(matrix="Rec601", interlaced=false)

Audiodub(WAVSource("R:\Testgraph.wav")).DelayAudio(-0.32).AssumeSampleRate(44100)

SyncAudio(0.9998472741694233)

smdegrain(tr=3,lsb=true,thsad=300,lsb_out=true,search=3,refinemotion=true)

Dither_convey_yuv4xxp16_on_yvxx()


I don't know why, there shouldn't be anything that can cause this.

I am using Avspmod avs2avi so the Audio (SyncAudio) is probably ignored, and even if it's not, that only stretches the audio, just a short function for it.

Other than that, it's only smdegrain, and it should be stable.

So i am wondering that it perhaps has something to do with Avisynth+?

The Crash is a simple: Microsoft Visual C++ Runtime Error

[SEt]

I seem to get crashing when i run this script, at least after a long time (Like 7 hours or something).
...
The Crash is a simple: Microsoft Visual C++ Runtime Error
Well, it doesn't tell much... At least process memory consumption at the crash time would help – it could be memory leak.

[zerowalker]

The memory consumption when i checked Task Manager and the Error was up for avs2avi was about 160mb if i remember correctly, far beyond a memory leak as far as i know.

EDIT:

Will run the script with Avisynth 2.6 to see if the results differ.
I can say that it's eating memory compared to Avisynth+, i think it will eventually crash, Avisynth+ stayed under 200mb as far as i remember, this is eating away and it's only been some minutes.

(Seems to have stayed at 589mb for now*)

[SEt]

With 2.6 you really need to play with SetMemoryMax if you are using it in MT mode, also huge memory consumption is normal if it's stable. I doubt testing in 2.6 would reveal anything interesting for Avisynth+ crash though.

[zerowalker]

With 2.6 you really need to play with SetMemoryMax if you are using it in MT mode, also huge memory consumption is normal if it's stable. I doubt testing in 2.6 would reveal anything interesting for Avisynth+ crash though.

I am not using MT, as it's unstable even on ST.
And well, better try to remove unneeded variables in the equation right?

However It didn't crash cause of a memory leak, however it did crash nevertheless.

Both crashes occur as pretty much the same place and time.

It's when there is 7:30 hours left (7:35 for Avisynth+, but it probably differ a bit per run).
The frame was 170481 and 170483 , this also probably differ.

The frames aren't corrupt and i can play them in Avspmod (or rather go step through them) without any problems.
So my guess is that it crashes around that time, and the frames doesn't matter.


This should however conclude that there isn't a Memory Leak (On crash 2.6 used about 421mb Ram on avs2avi).

So this must mean that something is wrong with smdegrain, SyncAudio and the other commands aren't used as far as i know.

EDIT:

I choose Debug when it crashed, so got it open in Visual Studio, however i have no idea about what to look for here, but will have it open if you have any clue what could be useful.

EDIT 2: Guess it doesn't yield anything so closed it.

[sl1pkn07]

hi,

i try to install r1689 by line0 (posted above) in my linux with wine

the instalation is ok, but when launch avs2pipe with any avs the avisynth/avs2pipe crash with this log

http://sl1pkn07.no-ip.com/paste/view/6d4ef6d6

with official r1576 works ok without any problem

any help?

greetings

[innocenat]

r1689 and possibly all future MT version of Avs+ (if there will ever be one) will not run on Wine due to, IIRC, VS2012 threading library. This is a wine's bug, not Avs+'s.

[sl1pkn07]

It is a known bug by the wine developers?

greetings

[qyot27]

Yes, and they've known for quite a long while:
https://appdb.winehq.org/objectManager.php?sClass=application&iId=5766
http://bugs.winehq.org/show_bug.cgi?id=17273

Also:
http://forum.doom9.org/showthread.php?p=1663971#post1663971

[sl1pkn07]

oh, thanks a lot

greetings

EDIT:
nope, don't work qyot27 workground with vcredist 2012 update4 extracted with WiX 3.6 (all other version fails/not run)

tsk

[SEt]

In case it might help with runtime installation problems for someone: AviSynthPlus-r1697.7z (https://www.dropbox.com/s/6q42kcsbrg5ss0p/AviSynthPlus-r1697.7z)

My semi-static build of current git with VS2013u2, no runtime requirement. Not tested, no idea if it even works.

[Overdrive80]

Hey, I am installing avs+ with installer and all fine. I want use multithread but function SetThreads() is not detected. How can I get this?? Thanks

[Groucho2004]

function SetThreads() is not detected
Never heard of it. If you want to try the multi-threaded test build, have a look at this post (https://forum.doom9.org/showthread.php?p=1666364#post1666364).

[Overdrive80]

Never heard of it. If you want to try the multi-threaded test build, have a look at this post (https://forum.doom9.org/showthread.php?p=1666364#post1666364).

Thanks Groucho2004, I didnt understand properly this post http://forum.doom9.org/showthread.php?p=1651085#post1651085, I thought that it was function and not a feature request.

[kaefert]

Hi there guys! I've seen that on page 27 of this thread the users dvdan23 and TurboPascal7 where discussing the compatibility of Avisynth+ with Wine under linux half a year ago. I wanted to ask if anybody knows anything about the current status of this. I've just tried to install AviSynthPlus-r1576.exe under a fresh wine context and it does not seem to work out of the box at least.

Running a version "script" gives this error and no picture:

$ WINEDEBUG="-all" wine avs2yuv "version.avs" - | mplayer -cache 2048 -noidx -
MPlayer2 2.0-701-gd4c5b7f-2ubuntu2 (C) 2000-2012 MPlayer Team

Playing -.
Reading from stdin...
Cache size set to 2048 KiB
Cache fill: 0.00% (0 bytes) wine: Call from 0x7b83aace to unimplemented function msvcr110.dll._libm_sse2_sqrt_precise, aborting

Avisynth error:
Evaluate: Unhandled C++ exception!
(version.avs, line 1)
Cache fill: 0.00% (0 bytes)

Failed to recognize file format.


Exiting... (End of file)

[qyot27]

http://forum.doom9.org/showthread.php?p=1684658#post1684658

And the reason is that the official builds of AviSynth+ are built with VS2012, and the 2012 runtime doesn't work under Wine. It's Wine's issue, which is up to them to resolve (they do try to implement this stuff on their own anyway to avoid requiring the Microsoft-provided runtimes; it's just a waiting game). Further, revisions of AviSynth+ after r1576 (not sure where the actual cut-off was, maybe in the 1630s?) can only be compiled by VS2012 or higher due to the use of C++11.

As far as r1576 is concerned, it has no such absolute restriction and can be built with VS2010. Said build would work under Wine. For those that need to do so, here is one such build:
http://www.mediafire.com/download/tthbb25kbn74bvk/avisynth+_r1576.zip



Also, for the love of all that's right in this world, switch to mpv (https://launchpad.net/~mc3man/+archive/ubuntu/mpv-tests) (PPA linked for feature completeness).

[sl1pkn07]

Hi

Official r1576 (https://github.com/AviSynth/AviSynthPlus/releases/download/Rel-r1576/AviSynthPlus-r1576.exe) runs for me out of the box

http://wstaw.org/m/2014/07/14/plasma-desktopu26031.png

(avs2yuv is launched by script in /usr/bin)

greetings

[kaefert]

thanks for the quick and very useful reply qyot27!

[kaefert]

Another question, is there maybe a 64bit build of Avisynth+ that has been built with VS2010 to work with wine?
I need 64bit to workaround this issue: http://forum.doom9.org/showthread.php?p=1687306
update: it doesn't need to be MT, but I need 64bit to workaround the only 28 source-files per script issue. (see link above)

[VideoFanatic]

Can someone please tell me how to get MT working with Avisynth+. I'm using DGDecodeNV to index my video. I'm using the Avisynth+ version of AvsPmod. Here's my script:


SetFilterMTMode("", 2)
SetFilterMTMode("DGSource", 3)
DGSource("E:\1 = Working Videos\WWE 2014\Raw from July\Raw Mar 24 2014.dgi")
AssumeTFF()
DeGrainMedian(limitY=2, limitUV=3, mode=1, interlaced=true)
Yadif(mode=1, order=1)
Vinverse()
Lanczos4Resize(1280,720)
Prefetch(4)


AvsPmod tells me there's no function named SetFilterMTMode

In normal Avisynth MT I just did this:
setmtmode(5,9)
DGSource("E:\1 = Working Videos\WWE 2014\Raw April - Dec\Raw Mar 24 2014.dgi")
setmtmode(2)

What's the equivalent with Avisynth+?

[Groucho2004]

AvsPmod tells me there's no function named SetFilterMTMode
You're using the wrong avisynth.dll. There is a post in this thread where ultim posted the MT DLL.

[VideoFanatic]

Can you give me a link to it please. I can't find it. It's not mentioned on the project's homepage or in the 1st post of this thread.

[Groucho2004]

Can you give me a link to it please. I can't find it. It's not mentioned on the project's homepage or in the 1st post of this thread.
http://forum.doom9.org/showthread.php?p=1666364#post1666364

[VideoFanatic]

OK I downloaded the MT version from that link and copied the dlls into the SysWow64 folder. When I start AvsPmod it says [Error126] The specified module could not be found. Loading avisynth.dll failed! Make sure that AviSynth is installed. Alternatively, specify now it's directory.

[Groucho2004]

OK I downloaded the MT version from that link and copied the dlls into the SysWow64 folder. When I start AvsPmod it says [Error126] The specified module could not be found. Loading avisynth.dll failed! Make sure that AviSynth is installed. Alternatively, specify now it's directory.
Did you install the "official" r1576 AVS+ version before? Also, you can check your Avisynth install with this tool (http://forum.doom9.org/showthread.php?t=170647).

[VideoFanatic]

Yes. Then I copied the MT dlls into the SysWow64 folder. Used the Avisynth Info Tool and it says I have Avisynth+ 0.1 (r1689, x86) installed.

[Groucho2004]

Yes. Then I copied the MT dlls into the SysWow64 folder. Used the Avisynth Info Tool and it says I have Avisynth+ 0.1 (r1689, x86) installed.
Did you try to open the script with a different program? VDub?

[VideoFanatic]

It seems to be working now. I forgot to restart my PC after installing Avisynth+ and copying the DLLs.

This was my non+ Avisynth MT script:

setmtmode(5,9)
DGSource("E:\Video.dgi")
setmtmode(2)
AssumeTFF()
DeGrainMedian(limitY=2, limitUV=3, mode=1, interlaced=true)
Yadif(mode=1, order=1)
Vinverse()
Lanczos4Resize(1280,720)


What is the equivalent of that speed in Avisynth+ MT? I tried the following which works but I've no idea if these are the correct settings or not?:

SetFilterMTMode("", 2)
SetFilterMTMode("DGSource", 3)

DGSource("E:\1 = Working Videos\WWE 2014\Raw from July\Raw Mar 24 2014.dgi")
AssumeTFF()
DeGrainMedian(limitY=2, limitUV=3, mode=1, interlaced=true)
Yadif(mode=1, order=1)
Vinverse()
Lanczos4Resize(1280,720)
Prefetch(9)

Can someone please tell me how MT works in Avisynth+ as I don't quite understand it. I presume the prefetch line is the number of threads you want but what do the underlined things mean?

[Groucho2004]

Can someone please tell me how MT works in Avisynth+ as I don't quite understand it.
All you have to do is read ultim's instructions in the post right below the one with the download link.
There is also a search function on this forum. One could for example search for the term "prefetch" in this thread. It's not that difficult.

[VideoFanatic]

I read them before I posted and I couldn't find anything via search. It doesn't explain what the "" part does. I'd like to know what the equivalent of the following is in Avisynth+:


setmtmode(5,9)
DGSource("E:\Video.dgi")
setmtmode(2)

Is the following script correct - everything is using mode 2?
You should use mode 2 unless mode 3 is required for the plugin to work? I want 9 threads so I entered Prefetch(9) at the end.

SetFilterMTMode("", 2)
SetFilterMTMode("DGSource", 2)

DGSource("E:\Video.dgi")
AssumeTFF()
DeGrainMedian(limitY=2, limitUV=3, mode=1, interlaced=true)
Yadif(mode=1, order=1)
Vinverse()
Lanczos4Resize(1280,720)
Prefetch(9)

[SamKook]

I'd suggest you read(again?) the comments from the example Ultim posted in the post below the one Groucho2004 linked, all you're asking is very well explained.

The SetFilterMTMode("", 2) line is to set the default MT mode to 2 so the "" means any filter.

There's no point in setting DGSource to 2 if you already set every filters that aren't explicitely specified to 2. But you should use 3 if you want the equivalent of 5 from avisynth and that is what Ultim suggested for most source filters.

As for the prefetch call, ultim very clearly says before the example "You enable MT by placing a single call to Prefetch(X) at the *end* of your script, where X is the number of threads to use." so yes, Prefetch(9) will give you 9 threads in MT mode(there was some things that weren't clear at one time on wether it used 9 threads for encoding or one was reserved for something else, but I don't remember exactly and I might be wrong. Either way, just try it and you'll see if it works fine).

[VideoFanatic]

Thanks, I understand now.

[Xebika]

I read them before I posted and I couldn't find anything via search. It doesn't explain what the "" part does. I'd like to know what the equivalent of the following is in Avisynth+:


setmtmode(5,9)
DGSource("E:\Video.dgi")
setmtmode(2)

Is the following script correct - everything is using mode 2?
You should use mode 2 unless mode 3 is required for the plugin to work? I want 9 threads so I entered Prefetch(9) at the end.

SetFilterMTMode("", 2)
SetFilterMTMode("DGSource", 2)

DGSource("E:\Video.dgi")
AssumeTFF()
DeGrainMedian(limitY=2, limitUV=3, mode=1, interlaced=true)
Yadif(mode=1, order=1)
Vinverse()
Lanczos4Resize(1280,720)
Prefetch(9)

What is the speed gain of your script with MT??

[thescrapyard]

Can anybody help in rewriting this dirt removal script that uses the original RemoveGrain plug-in and also mvtools2, so that it uses AviSynth+ RGTools instead and what looks like an improved clense function according to the RGTools wiki

Clense (clip, clip "previous", clip "next", bool "grey")



I really wouldn't know where to start, I've stared at the script but not knowing what functions to change


function RemoveDirt(clip input, int limit, bool _grey)
{
clensed=input.Clense(grey=_grey, cache=4)
alt=input.removegrain(2)
return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=4,cthreshold=6, gmthreshold=60,dist=2,dmode=2,debug=false,noise=limit,noisy=12,grey=_grey)
}

function RemoveDirtMC(clip,int limit, bool "_grey")
{
_grey=default(_grey, false)
limit = default(limit,6)
bvec = clip.MVAnalyse(isb=false, blksize=8, delta=1, pel=2, sharp=1, truemotion=true, idx=4, overlap=0)
fvec = clip.MVAnalyse(isb=true, blksize=8, delta=1, pel=2, sharp=1, truemotion=true, idx=4, overlap=0)
backw = clip.MVFlow(bvec, idx=4)
forw = clip.MVFlow(fvec, idx=4)
clp=interleave(backw,clip,forw)
clp=clp.RemoveDirt(limit,_grey)
clp=clp.SelectEvery(3,1)
return clp
}

This may also help others that want to replace functions or plug-ins from the original AviSynth plug-ins or functions, especially when trying to use 32-bit and 64-bit AviSynth+

[innocenat]

I don't know about other, but I don't think it is right topic to ask.

For RgTools, you can just replace removegrain[something].dll with just rgtools dll (don't forget to remove old dll), the new version will be use automatically as they provide same functions as old version. I don't know about mvtools2, but documentation suggest the you can just change MV*** to M*** and it will works.

Please also note that those changes aren't related to Avisynth+ in anyway, as all will also work with original avisynth 2.6alpha5. For 64-bit Avisynth+ just use 64bit version of the dll, the rest is the same.

[feisty2]

here you go
function RemoveDirt(clip input, int "limit", bool "_grey")
{
clensed=input.Clense(grey=_grey)
alt=input.removegrain(2)
return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=4,cthreshold=6, gmthreshold=60,dist=2,dmode=2,debug=false,noise=limit,noisy=12,grey=_grey)
}

function RemoveDirtMC(clip clip,int "limit", bool "_grey")
{
_grey=default(_grey, false)
limit = default(limit,6)
super = clip.msuper (sharp=1,pel=2)
bvec = super.MAnalyse(isb=false, blksize=8, delta=1, truemotion=true, overlap=4)
fvec = super.MAnalyse(isb=true, blksize=8, delta=1, truemotion=true, overlap=4)
backw = clip.MFlow(super, bvec)
forw = clip.MFlow(super, fvec)
clp=interleave(backw,clip,forw)
clp=clp.RemoveDirt(limit,_grey)
clp=clp.SelectEvery(3,1)
return clp
}
"RestoreMotionBlocks" function is from here http://www.removedirt.de.tf/

[Reel.Deel]

here you go


RgTools' Clense does not have the 'cache' parameter.

[feisty2]

I already corrected it, thank you for tipping

[bennynihon]

I have an 8 core i7 with 16GB of RAM running x64 Windows 8.1. For whatever reason feeding x264 my Avisynth+ MT script crashes with Prefetch > 4. Prefetch(4) mostly works, but here's the kicker. My script with one clip works just fine with Prefetch(4), while another source clip using the same exact script crashes with Prefetch(4). I need to use Prefetch(3) for that one. I notice that avisynth and x264 use comparatively very little memory. About 800MB each. I'm OK with leaving it at Prefetch(4) as the performance is satisfactory to me. But I'm concerned by the lack of consistency even with the same script. Are there any other tweaks or fine tuning I can do to possibly get better consistency or performance? Here's my script:

SetFilterMTMode("", 2)
SetFilterMTMode("FFVideoSource", 3)

# Video Source
FFVideoSource("video.mkv")

# Crop
Crop(0,20,0,-20) # 1920x1040 (1.85:1 AR)

# Masks
vLumaMask=mt_binarize(threshold=64, upper=true).grayscale().BilinearResize((width/32)*2, (height/32)*2).BilinearResize(width,height).mt_binarize(threshold=254)
vEdgeMask=mt_edge(mode="prewitt",thy1=0,thy2=16).grayscale().mt_binarize(threshold=16, upper=true).BilinearResize((width/32)*2, (height/32)*2).BilinearResize(width,height).mt_binarize(threshold=254)
vGrainMask=mt_logic(vLumaMask,vEdgeMask,mode="and")
vDegrainMask=vGrainMask.mt_invert()

# Degrain
bs = 16
super = MSuper(pel=2,sharp=1,hpad=bs,vpad=bs)
bvec2 = MAnalyse(super, isb=true, delta=2, blksize=bs, overlap=bs/2)
bvec1 = MAnalyse(super, isb=true, delta=1, blksize=bs, overlap=bs/2)
fvec1 = MAnalyse(super, isb=false, delta=1, blksize=bs, overlap=bs/2)
fvec2 = MAnalyse(super, isb=false, delta=2, blksize=bs, overlap=bs/2)
vClean=MDegrain2(super,bvec1,fvec1,bvec2,fvec2,thSAD=400)
mt_merge(vClean,vDegrainMask,u=2,v=2)

# Dither
Dither_convert_8_to_16()
vGrain=Dither_add_grain16(var=0.6,uvar=0,lsb_in=true)
Dither_merge16_8(vGrain,vGrainMask,u=2,v=2)

# Output
DitherPost(mode=3) # 8-bit output

# Multi-threaded
Prefetch(3)

[Dion]

I cannot seem to get AvsPMod to display video with Avisynth+. It always locks up when I try to load a mkv into it. Does it from DSS2, FFVideoSource, DGSource. All of them.

I have the latest Avisynth+ Installed. I am using the AvsPMod recommended on the Avisynth+ Page. If I use the vanilla AvsPMod 2.5.1 I get an error at startup. And some plugin's don't work. But I am able to display video.

Does anyone know why?

[thescrapyard]

Try this, I use avisynth+ with avspmod as well

http://www6.zippyshare.com/v/49944794/file.html


This link is from the avisynth+ site and specifies this version of avspmod that has been tested as ok with avisynth+

Also, try telling avisynth/avisynth+ where to find plugins and scripts. Its good at falling over with obscure errors if it can't find what its looking for. I've done it this way for years with no issues since it was suggested to me.... and use MKV as well

This is a couple of lines from scripts I use with avspmod


dss2("g:\ConvertXToDVD\dangermouse.mkv")
ConvertToYV12()

DLLPath="C:\Program Files (x86)\Applications\--Scripts\"

I just use the DLLPath variable as it then means i don't have a messy script and makes life easier, like this. If I miss out the ConverttoYV12 it fails with an error, so this just makes sure its ok


dss2("g:\ConvertXToDVD\dangermouse.mkv")
ConvertToYV12()
DLLPath="C:\Program Files (x86)\Applications\--Scripts\"

SetFilterMTMode("",2)

Loadplugin(DLLPath+"-denoiser\mvtools2.dll")
Loadplugin(DLLPath+"-denoiser\removedirt.dll")
Loadplugin(DLLPath+"-denoiser\removegrain.dll")
Loadplugin(DLLPath+"-denoiser\repair.dll")
Import(DLLPath+"-denoiser\removedirtmc.avs")

removedirtmc(75,false)

prefetch(3)

[Dion]

Try this, I use avisynth+ with avspmod as well

http://www6.zippyshare.com/v/49944794/file.html

I already said I used it in my previous post.

Edit: I have found the problem nevermind.

[jmartinr]

I already said I used it in my previous post.

Edit: I have found the problem nevermind.

What was the problem/solution?

[Dion]

What was the problem/solution?

One of my plugin's doesn't like Avisynth+. Deleted it and it all worked fine. I msged the dev of the plugin to fix it. So hoping he does.

[StainlessS]

One of my plugin's
Is the name of the plugin a secret ?

[Octo-puss]

Does this project have any chance of ever being resurrected? Right now it feels like so much wasted potential.

[aegisofrime]

Does this project have any chance of ever being resurrected? Right now it feels like so much wasted potential.

It's funny that Avisynth has so much history of promising projects being dropped. As I mentioned earlier in the thread, the history of Avisynth+ is uncannily like that of Avisynth x64. Both started out with great promise with the developer showing great enthusiasm and developing the project with great speed. Then they disappear suddenly :scared:

Don't get me wrong, I don't blame them or anything. They are giving their time away for free and for that I'm appreciative. I'm just curious what is it that cause them to disappear in that manner. :(

[TurboPascal7]

Does this project have any chance of ever being resurrected? Right now it feels like so much wasted potential.
ultim is alive and might return to the project in the future. I wouldn't expect other people to join him though.

I'm just curious what is it that cause them to disappear in that manner. :(
Avisynth development is one of the least rewarding activities for a programmer and the whole idea of complicated and powerful frameservers is dying, imho. After a few months most of the "fun" wears off and all you left with is extremely boring maintenance, depressing codebase, tiny community and zero motivation to continue spending your time on it.

Something like that.

[macromizer]

It's funny that Avisynth has so much history of promising projects being dropped. As I mentioned earlier in the thread, the history of Avisynth+ is uncannily like that of Avisynth x64. Both started out with great promise with the developer showing great enthusiasm and developing the project with great speed. Then they disappear suddenly :scared:

Don't get me wrong, I don't blame them or anything. They are giving their time away for free and for that I'm appreciative. I'm just curious what is it that cause them to disappear in that manner. :(

As said by TurboPascal, it's just a huge drudge trying to fixup and maintain a legacy codebase you had no hand in originally writing. Avisynth development before ultim came along also suffered a lot from the same perception of ffmpeg before the libav fork. That perception being that the maintainer(s) are slow to accept patches, if at all, and a resistance to a lot of changes necessary to bring in more contributors (in the case of Avisynth requiring maintaining compatibility with a 16-year-old version of Visual Studio). While ffmpeg became much quicker to accept patches and new features after being forked, Avisynth unfortunately just maintained the status quo.

And then the Avisynth 3 guys simply just bit off way more work than they could reasonably do with only a couple of people at most. And unfortunately if ultim is going to end up being the only driving force for Avisynth+ it's going to suffer the same fate.

Even in the worst scenario for Avisynth/Avisynth+, Vapoursynth development seems to be picking back up with more contributors than just Myrsloik and more filters being ported over.

I was looking into porting Avisynth+ over to OS X and possibly after that to iOS (simply for the challenge), but I got stalled and am even wondering if it's even something worthwhile considering that quite a bit of porting effort would be needed to get a decent amount of plugins in addition to the modernization that TP and others have already done and the fact that a number of useful yet closed-source plugins won't ever run natively. And if you've already got to port things, I might as well just go with Vapoursynth. I would be more than willing to look back into it if there was a genuine want for it, but being somewhat fatalistic it seems to be a futile effort for little gain over something already up and running.

[qyot27]

Yes, because a pause of 5 months (two months if you count the pull from June) is in any way comparable to a 5+ year, prolonged development hell. /sarcasm

Think about it for a second: within a time span of 6 months (September 2013-March 2014), there were enough unique commits to AviSynth+ to outpace the entire development cycle of AviSynth 2.6 up to that point, and even a chunk of 2.5's. It's not unreasonable, especially with the volume of different things that were getting done (64-bit support, intrinsics, MT & caching, etc.), to expect some level of burn-out or slow down. And even granted that, the 'level' that avsplus is currently at is not in any way lagging behind 2.6 - which hasn't had any commits since September 2013 when 2.6a5 was released.


I was looking into porting Avisynth+ over to OS X and possibly after that to iOS (simply for the challenge), but I got stalled and am even wondering if it's even something worthwhile considering that quite a bit of porting effort would be needed to get a decent amount of plugins in addition to the modernization that TP and others have already done and the fact that a number of useful yet closed-source plugins won't ever run natively. And if you've already got to port things, I might as well just go with Vapoursynth. I would be more than willing to look back into it if there was a genuine want for it, but being somewhat fatalistic it seems to be a futile effort for little gain over something already up and running.
I actually saw this on Github back when it was pushed, and was looking to integrate the OSX changes and innocenat's Linux work together into a single branch. I certainly would be willing to assist on tedious work (I did transition the English documentation over to RST, even though the content needs updating). I started trying to do this in such a way to get it building on Linux with GCC, but I hit a snag in avisynth.cpp and I've been procrastinating on trying to look for solutions (or jumping on IRC to do the discussion in real time, since most of the time I use IRC under Windows but do any compilation tests and such under Linux). One particularly nasty issue, is that GCC's handling of intrinsics for CPU runtime detection is braindead and requires each instruction set's intrinsics to be segregated into separate files, because otherwise, you'll only ever be able to compile the project for the highest instruction set used, completely defeating the purpose. CMake is capable of handling this situation (x265 was set up this way for a while, before they moved [virtually?] everything to YASM), but it requires special-casing so that the necessary -m flags for the intrinsics are applied only on a per-file basis. This was most of what I was doing before I hit that roadblock, but I hadn't set up the stuff for CMake to deal with it yet.

IMO, the value of detethering it from Windows is in getting AvxSynth out of the way - that's what I'm really most concerned about right now. It also would help to expand the potential developer base, as every so often there's a new pull request or issue opened over on AvxSynth's repo that I can practically guarantee will come to nothing. The lasting effect AvxSynth had was to finally get libavformat's AviSynth demuxer rewritten properly, but AvxSynth itself is not a long-term solution (and it's at a major feature disadvantage, because it's based on 2.5 with a few various backports from 2.6, doesn't support the newer colorspaces from 2.6, and has all its assembly disabled). Having something that's properly up-to-date with the development HEAD should go a long way toward getting more use on non-Windows platforms, and then we might actually see more plugins ported over or new plugins getting written for it. I don't place a lot of emphasis on the state of the plugins to determine the value of getting the core working there, since even the core in a vacuum has features that are useful on their own (most of my DVD authoring workflow relies on scripts only using core functions, apart from using FFMS2 to load the stuff). There was also an offhanded comment by IanB on the AvxSynth thread that since most plugins are simply algorithm code, it might be possible to thunk them - which would at least be a band-aid until they're ported properly (this is basically what mplayer-svn does with w32codecs/w64codecs, right?).



Maybe if we go about it like, "okay, for this development cycle we're going to focus on this feature", and then lay out the actual steps - in public view on this thread - on how to achieve that, with technical details, it might help to motivate everyone (and actually give users more insight into what's going on so we don't have all the easily demotivating 'is it dead or not?' talk getting posted). That's one thing that I think is contributing to the drag, since many of the discussions about how to go about implementing new features would be relegated to IRC and this thread is basically for people to ask about how to use MT (lately, anyway). It would be one thing if the IRC logs were cached somewhere and downloadable/viewable online for reference, but as far as I know, they aren't. One thing for instance, is how to translate from MSVC-isms to something that all three of the common compilers (MSVC, GCC, and Clang) can digest, where the gotchas actually are, and in those cases that the dialect is unavoidable, what the equivalent is in the others so that the branching can be set up cleanly.

I haven't really seen any more big reports of MT failing in the months since it appeared (in terms of its actual stability), save for the bounds checking issue with the audio cache that I reported in here and on Github. If I'm correct about that assessment, perhaps it's time to just bite the bullet and merge it to master and bump the version number to 0.2. The psychological effect of just going ahead and releasing it might help too.

[Octo-puss]

and the whole idea of complicated and powerful frameservers is dying, imho
Hm, but what else are we supposed to use instead of Avisynth if it's dying then?

[macromizer]

Yes, because a pause of 5 months (two months if you count the pull from June) is in any way comparable to a 5+ year, prolonged development hell. /sarcasm

Think about it for a second: within a time span of 6 months (September 2013-March 2014), there were enough unique commits to AviSynth+ to outpace the entire development cycle of AviSynth 2.6 up to that point, and even a chunk of 2.5's. It's not unreasonable, especially with the volume of different things that were getting done (64-bit support, intrinsics, MT & caching, etc.), to expect some level of burn-out or slow down. And even granted that, the 'level' that avsplus is currently at is not in any way lagging behind 2.6 - which hasn't had any commits since September 2013 when 2.6a5 was released.

If this was towards me, I would definitely agree that the two are not analogous situations and I wasn't meaning to make it seem that. Avisynth+ with what ultim and others have done has been amazing in kickstarting Avisynth activity again and what they have done is pretty awesome. My point is that I just worry that if everyone else ends up dropping out before ultim comes back that it's just going to suffer the same fate as IanB driving Avisynth alone. It's just so easy to burn out if you're a one-man development band.

[innocenat]

Regarding Linux/OSX/GCC/LLVM port, qyot27 is very spot on. Even then, because of how ssse3/sse4.1 code are interleaved with sse2, it is even quite more work than other project. I have been thinking about this for quite a while, bur haven't start yet.

Ultim is alive and does reply to questions on IRC. If he has more free time I am quite sure he will continue developing. I also plan to continue developing, I just don't have enoigh free time and experience to handle the project myself.

[Seedmanc]

Well, at least Avisynth x64 works, the parts that were ported, that is.

Hm, but what else are we supposed to use instead of Avisynth if it's dying then?
I second this question. It can only die if it were to be replaced by something more effective at solving same tasks, but I can't imagine doing stuff without Avisynth, neither VDub nor Vegas-like editors fill its niche.

[qyot27]

If this was towards me, I would definitely agree that the two are not analogous situations and I wasn't meaning to make it seem that. Avisynth+ with what ultim and others have done has been amazing in kickstarting Avisynth activity again and what they have done is pretty awesome. My point is that I just worry that if everyone else ends up dropping out before ultim comes back that it's just going to suffer the same fate as IanB driving Avisynth alone. It's just so easy to burn out if you're a one-man development band.
No, it wasn't directed at you. It was directed at the 'there's not been an update in a month, so is it dead now? I guess I should move back to 2.6' types of responses from general users...because it started happening not *that* long into the slowdown in activity.

Even then, because of how ssse3/sse4.1 code are interleaved with sse2, it is even quite more work than other project.
Do you mean there's SSSE3/SSE4.1 stuff in the functions with _sse2 suffixes in their names? Or just that all the different functions are all packed into the same source file? I assume it's the latter.

The thing I was getting confused about were the functions suffixed with _ssex and _xsse. I wasn't sure where to put those when splitting the files in convert/.

[feisty2]

Hm, but what else are we supposed to use instead of Avisynth if it's dying then?

imho, avisynth itself might not be the best up till now, but there are so many magic filters and scripts, thats what makes it so irreplaceable

[innocenat]

Do you mean there's SSSE3/SSE4.1 stuff in the functions with _sse2 suffixes in their names? Or just that all the different functions are all packed into the same source file? I assume it's the latter.

The thing I was getting confused about were the functions suffixed with _ssex and _xsse. I wasn't sure where to put those when splitting the files in convert/.

The SSE3 lddqu, SSSE3 pshufb etc are injected by templated parameter into sse2 core function, to reduce code duplication.

I think the core and templated function should be put in #ifdef in header file and include into sse2/3/4 file, but I am not sure if this will work or not.

[foxyshadis]

Well, at least Avisynth x64 works, the parts that were ported, that is.


I second this question. It can only die if it were to be replaced by something more effective at solving same tasks, but I can't imagine doing stuff without Avisynth, neither VDub nor Vegas-like editors fill its niche.

ffmpeg's filtering is becoming much more capable, with a lot of ideas taken from Avisynth filters. Meanwhile, real-time filtering with multiple shaders is now easy during playback, even with low-end GPUs. I don't intend to stop using Avisynth anytime soon, but the writing is on the wall.

[TurboPascal7]

Hm, but what else are we supposed to use instead of Avisynth if it's dying then?

Nothing. Hth, bye.

No, seriously, you don't have to use anything instead of avisynth. Why would you do this? It works for what it was designed for and won't suddenly break tomorrow.

I'm just saying that the whole idea of extremely complex and powerful frameservers is kinda useless these days, when most video requires light processing, that can be done in real-time on playback, if any at all.

[Octo-puss]

I thought Avisynth was used for encoding movie rips somehow? Please don't laugh at me, I don't understand this stuff one bit. I just use MeGUI and it depends on the old single threaded Avisynth, which is painfully slow when using QTGMC (waiting 3 days to encode something on i7 is just.... uh).

[Groucho2004]

I just use MeGUI and it depends on the old single threaded Avisynth, which is painfully slow when using QTGMC (waiting 3 days to encode something on i7 is just.... uh).
I don't think megui is restricted to one Avisynth version. I'm sure you can use either SEt's MT or AVS+ MT.

3 days with an i7 is insane, even with single threaded processing. I'd like to see that script.

[Octo-puss]

I don't think megui is restricted to one Avisynth version. I'm sure you can use either SEt's MT or AVS+ MT.

3 days with an i7 is insane, even with single threaded processing. I'd like to see that script.

No, it's not, but the multithreaded version is horribly bugged and the whole process crashes after several hours unpredictably.
I haven't tried Avisynth+ as it's unfinished.

Yes, that's the downside of QTGMC. It takes a LONG time with single thread. That's why I was hoping Avisynth+ would replace Avisynth.

[Boulder]

Multithreading QTGMC scripts is tricky, sometimes it seems impossible to make it through the whole video without crashing.

As an alternative, you could try using the MVTools version by cretindesalpes (inside the Dither package). It is internally multithreaded and runs faster.

[macromizer]

No, it wasn't directed at you. It was directed at the 'there's not been an update in a month, so is it dead now? I guess I should move back to 2.6' types of responses from general users...because it started happening not *that* long into the slowdown in activity.

Okay. :)

Well after I finish up a couple other projects I've been meaning to finish for more than a year now, I'll look back into possibly doing some more OS X porting work. I'll definitely get in contact if I decide to end up working on it again.

[Groucho2004]

As an alternative, you could try using the MVTools version by cretindesalpes (inside the Dither package). It is internally multithreaded and runs faster.
Never tried this before, very nice speed-up!

[foxyshadis]

No, it's not, but the multithreaded version is horribly bugged and the whole process crashes after several hours unpredictably.
I haven't tried Avisynth+ as it's unfinished.

It's not set's MT or avs+ that's bugged (well, not entirely), it's actually mvtools itself. It's fairly buggy, but a lot of the bugs are only exposed during parallel processing, so you get most of these crashes on slow mvtools scripts where you need the speed the most of all.

It'd be a bit painful but quite possible to create a version of QTGMC that used avs threading for all filters but mvtools, where only the internally threaded version works correctly right now, if anyone is interested. It's past due for a new plugin pack with the improved avs+ versions of filters, too.

Myrsloik is wrestling with that one in his copious free time.

[jpsdr]

Yes, that's the downside of QTGMC. It takes a LONG time with single thread.

One trick is to split your file in several parts and process each part in the same time. You'll keep QTGMC single threaded, but take advantage of several CPU. The best would be that each filter used has internal MT, but it seems it's not the case.

[Dion]

Is the name of the plugin a secret ?

Nope its called SupTitle. It's for burning sup subtitles into a video.

[Elegant]

Any idea what can cause MeGUI or AvsPMod to hang when using AviSynth+? I'm using Windows 7 x64 and even loading a video using any method (DGDecode, FFMS2) in a script causes both programs to hang (only occurs with AviSynth+). Encoding is fine though but I'd prefer to actually see a preview >.>

[Groucho2004]

Any idea what can cause MeGUI or AvsPMod to hang when using AviSynth+? I'm using Windows 7 x64 and even loading a video using any method (DGDecode, FFMS2) in a script causes both programs to hang (only occurs with AviSynth+). Encoding is fine though but I'd prefer to actually see a preview >.>

Which version of AVS+ are you using? 64 or 32 Bit?
Be aware that 64 Bit Avisynth cannot load 32 Bit plugins (like DGDecode) and vice versa.
Define "hang". No error message?
Post your script.

[Elegant]

32 bit version; the 64 bit version shouldn't load with MeGUI nor AvsPMod (this is also version listed on the AviSynth+ website) as far as I know. You are right in assuming there is no error, it simply attempts to open and hangs on both programs. Script examples:

LoadPlugin("C:\Program Files (x86)\MeGUI\tools\dgindex\DGDecode.dll")
DGDecode_mpeg2source("E:\Output\VTS_01_4.d2v", info=3)

LoadPlugin("C:\Program Files (x86)\MeGUI\tools\ffms\ffms2.dll")
FFVideoSource("L:\MOVIE\BDMV\STREAM\00508.m2ts", cachefile="E:\Output\Movie.ffindex", fpsnum=24000, fpsden=1001, threads=1)

Both cause the issue. Both plugins are 32 bit as that's what MeGUI uses.

[qyot27]

I can encode just fine using x264 but I would prefer the ability to see what my work will look like instead of encoding blind.
VirtualDub? mpv?


And for obviousness' sake, you did already run the video through DGIndex or ffmsindex, right? The .d2v/.ffindex files do exist, is what I'm saying.

[Elegant]

VirtualDub? mpv?


And for obviousness' sake, you did already run the video through DGIndex or ffmsindex, right? The .d2v/.ffindex files do exist, is what I'm saying.

I'm talking command line x264 it works just fine. Yes, I did run them through those 2, one is a DVD and the other is a Bluray. The .d2v/.ffindex were made before I installed AviSynth+ though but it should be unrelated. I can open the files just fine for previewing with 2.6.0a5 MT but AviSynth+ just doesn't play nice for some reason. I was hoping to take advantage of some of AviSynth+'s features once I could actually load a video but so far even running those import lines by themselves causes both programs to hang.

[qyot27]

And if you try opening the scripts in VirtualDub and mpv?

[Elegant]

Unsure of what mpv is but using VirtualDub 1.10.4 also causes it to hang on open. Mind you VirtualDub wasn't able to actually load the DGDecode plugin for that test case but for FFMS2 it hangs.

[qyot27]

Mind you VirtualDub wasn't able to actually load the DGDecode plugin for that test case but for FFMS2 it hangs.
VirtualDub doesn't load AviSynth plugins. Was there an error message when you tried opening the .d2v script?


mpv is a media player:
http://mpv.io

Like most/all mplayer variants, it's typically started from the command line:
mpv test.avs

If the script won't play, copy/paste the error messages here.



How are you switching between the 2.6 MT and AviSynth+ dlls?
How did you install AviSynth+? If it was from the .7z file, did you also install the MSVC 2012 Runtime?
Put DGDecode and FFMS2 in AviSynth+'s plugins folder and remove the LoadPlugin lines from the scripts - does that do anything?

[foxyshadis]

Try Process Explorer, as well. You can open up the process and look at the threads, where you might see which plugin is the one that isn't initializing properly. If it's not using any CPU, that probably means you hit a deadlock, though, which would be weird to see consistently. If all the cpu is in avisynth.dll itself, then you might have to get Visual Studio Express and compile it yourself with debugging on to find out why (ugh), since the Avisynth+ installers don't include debugging info.

[Elegant]

Found it, AvsFilterNet.dll is the trouble maker. I don't think I've used any plugins that have relied on it so I guess I don't really need it.

[asarian]

Use installer from the homepage. If classic AviSynth is already installed, you will be offered the choice of either upgrading or replacing it. In the first case you'll be able to switch back to classic AviSynth by simply uninstalling AviSynth+. Your existing plugins will be kept in both cases.

Hmm, any way I can use my official 32-bit 2.6.0 AviSynth *next* to the 64-bit version of this one? I recall from 2.5.8, that you could use both 32 and 64-bit versions next to each other.

[Groucho2004]

Hmm, any way I can use my official 32-bit 2.6.0 AviSynth *next* to the 64-bit version of this one?
Sure. Install both AVS+ versions (32 & 64) using the installer from avs-plus.net.
Replace the avisynth.dll and the devil.dll in SysWow64 with the files from this archive (https://www.dropbox.com/s/qxj7z43m4y05z9g/AVS260_ICL.zip?dl=0).

[asarian]

Sure. Install both AVS+ versions (32 & 64) using the installer from avs-plus.net.
Replace the avisynth.dll and the devil.dll in SysWow64 with the files from this archive (https://www.dropbox.com/s/qxj7z43m4y05z9g/AVS260_ICL.zip?dl=0).

Thx. :) That's a neat trick. It *will* require me to give up on the official AS, though, right? Reason I wanted to install a parallel 64-bit version, is so I can keeping using MT for the 32-bit version.

[Groucho2004]

It *will* require me to give up on the official AS, though, right? Reason I wanted to install a parallel 64-bit version, is so I can keeping using MT for the 32-bit version.
If you want SEt's 32 Bit MT version, just use that DLL instead of the one I put in the zip file.

Edit: The only "official" version is the one on Sourceforge. All other flavours (SEt's MT, my ICL build, AVS+) can be used and are probably just as stable in most cases.

[feisty2]

Thx. :) That's a neat trick. It *will* require me to give up on the official AS, though, right? Reason I wanted to install a parallel 64-bit version, is so I can keeping using MT for the 32-bit version.

both SEt's mt version and avs+ are improved (but not official) versions, the official avs 2.6 is IanB's version

[asarian]

If you want SEt's 32 Bit MT version, just use that DLL instead of the one I put in the zip file.

Thx again. :) Will do so.

[feisty2]

Thx. :) That's a neat trick. It *will* require me to give up on the official AS, though, right? Reason I wanted to install a parallel 64-bit version, is so I can keeping using MT for the 32-bit version.

u can enjoy mt benefits on avs x64 also, search "mp_pipeline" on forum, the tool gets u mt functions for both ia32 and x64 versions of avs

[asarian]

u can enjoy mt benefits on avs x64 also, search "mp_pipeline" on forum, the tool gets u mt functions for both ia32 and x64 versions of avs

That's great news! Thx! :)

[asarian]

If you want SEt's 32 Bit MT version, just use that DLL instead of the one I put in the zip file.

Edit: The only "official" version is the one on Sourceforge. All other flavours (SEt's MT, my ICL build, AVS+) can be used and are probably just as stable in most cases.

Hehe; since I was already using MT dll's, I guess I never really used to the official package to begin with. :o

[qyot27]

both SEt's mt version and avs+ are improved (but not official) versions, the official avs 2.6 is IanB's version
Technically, only SEt's would be a non-'official' version of 2.6, because it's not really diverged much from classic. AviSynth+ is a separate project that was forked from 2.6, but has diverged significantly to the point that it's just not accurate to label it an 'unofficial' version of 2.6. It'd be like saying GNU and Linux are an 'unofficial' version of BSD.

Although for that matter, anyone that takes a look at AviSynth's development history will see very quickly that things have always been a bit scattershot with the main development. The developers involved often change a lot between the different trees that sprouted up - the original BRG version, the patched versions in the 1.0 series after BRG left (various authors), the development on 2.0 and 2.5.0-2.5.5 (sh0dan, et al.), 2.5.6-2.5.8 and 2.6 (increasingly IanB) are all technically distinct dev branches - 2.0-2.6 simply had the benefit of a single main repository, and just changed lead developers. Had AVS3.0 not died back in 2007, it would have exhibited the same properties - the devs in charge of it were different too, and the SVN repo for it was separate from the CVS repo for 2.0-2.6.

2.5/2.6 only coalesced as 'official' because it became widely used and was updated consistently for a few years, not because the devs that worked on it have a steering committee that gets to decide what 'official' is over the entire AviSynth ecosystem, only in the context of their own branches. AviSynth+ is hardly a deviation from that, and has its own 'official' (or more accurately, 'stable') builds - currently 0.1 r1576, with 'unofficial' ('dev') builds from the main/MT branch, which is up to r1699.

[feisty2]

well, thx qyot27 :), got it, I never really dug much into avisynth history

[asarian]

Technically, only SEt's would be a non-'official' version of 2.6, because it's not really diverged much from classic. AviSynth+ is a separate project that was forked from 2.6, but has diverged significantly to the point that it's just not accurate to label it an 'unofficial' version of 2.6. It'd be like saying GNU and Linux are an 'unofficial' version of BSD.

Although for that matter, anyone that takes a look at AviSynth's development history will see very quickly that things have always been a bit scattershot with the main development. The developers involved often change a lot between the different trees that sprouted up - the original BRG version, the patched versions in the 1.0 series after BRG left (various authors), the development on 2.0 and 2.5.0-2.5.5 (sh0dan, et al.), 2.5.6-2.5.8 and 2.6 (increasingly IanB) are all technically distinct dev branches - 2.0-2.6 simply had the benefit of a single main repository, and just changed lead developers. Had AVS3.0 not died back in 2007, it would have exhibited the same properties - the devs in charge of it were different too, and the SVN repo for it was separate from the CVS repo for 2.0-2.6.

2.5/2.6 only coalesced as 'official' because it became widely used and was updated consistently for a few years, not because the devs that worked on it have a steering committee that gets to decide what 'official' is over the entire AviSynth ecosystem, only in the context of their own branches. AviSynth+ is hardly a deviation from that, and has its own 'official' (or more accurately, 'stable') builds - currently 0.1 r1576, with 'unofficial' ('dev') builds from the main/MT branch, which is up to r1699.

Learn something every day. :) Thx.

[chainik_svp]

Is it dead or alive? o_O
In other words, if I think I found a bug - should I try to fix it myself or post here and wait for support?

===
Short version of issue is: x64 MT version of AVS+ doesn't free memory so if I want to run some memory consuming script inside MPC-HC x64 + ffdshow x64 it will eat some amount of memory on each seek.


enter simple script into ffdshow:
SetFilterMTMode("",2)
SetFilterMTMode("ffdShow_source",3)
ffdShow_source()
BilinearResize(3000,1500)
BilinearResize(1920,1080)
Prefetch(10)


and then just check and uncheck "Avisynth" check box while looking at Task Manager memory graph

===
and it's not an issue with 32-bit version

[ryrynz]

Is it dead or alive? o_O
In other words, if I think I found a bug - should I try to fix it myself or post here and wait for support?


Pretty much dead. I wouldn't rely on anyone getting to this in a timely fashion, if you're able to work on it go for it IMO.

[innocenat]

Is it dead or alive? o_O
In other words, if I think I found a bug - should I try to fix it myself or post here and wait for support?

===
Short version of issue is: x64 MT version of AVS+ doesn't free memory so if I want to run some memory consuming script inside MPC-HC x64 + ffdshow x64 it will eat some amount of memory on each seek.


enter simple script into ffdshow:
SetFilterMTMode("",2)
SetFilterMTMode("ffdShow_source",3)
ffdShow_source()
BilinearResize(3000,1500)
BilinearResize(1920,1080)
Prefetch(10)


and then just check and uncheck "Avisynth" check box while looking at Task Manager memory graph

===
and it's not an issue with 32-bit version

Do you have revision number/where did you get the build? If it's resizer issue the I can look into it, but no promise since ultim isn't here to release it.

[chainik_svp]

It's not a resizer issue, BilinearResize() is used just as an example to illustrate it's not a some filter issue but AVS+ in general.
Still it can be not AVS+ x64 issue but ffdshow x64 but I prefer to think that memory allocated by AVS+ should be freed by AVS+.

where did you get the build?

latest sources from MT branch built by me :)

[chainik_svp]

[for every frame]
{
VideoFrame *frame = it->second;
if (frame->vfb->refcount == 0)
delete frame->vfb;

delete frame;
}

why "if (frame->vfb->refcount == 0)" ??
vfb->refcount decremented in VideoFrame destructor so it can be (and it IS) >0
and ~VideoFrame doesn't delete vfb but just releases it

=====
Fixed it, see https://github.com/AviSynth/AviSynthPlus/issues/49

Is there any chance to include this fix to the main branch and release a new official build?

[foxyshadis]

Ouch. That makes sense, now I know why I was getting out-of-memory when testing some script variations last month.

[chainik_svp]

this isn't all the truth :D

if the filter was instantiated - it'll never be deleted...

in other words if you'll write some filter doing something like this:

SomeFilter::SomeFilter()
{ buffer = new BYTE[1000000000]; }

SomeFilter::~SomeFilter()
{ delete[] buffer; }

than you'll end with 1 GB of memory stolen cause it'll never be freed :)

[qyot27]

My question is: why is this specific to 64-bit, though? The snippet doesn't look specialized to me (noting, of course, I have no real knowledge of what you have to do for 64-bit vs. 32-bit in the code itself).

What I'm getting at is, is there any chance that it's caused by being miscompiled? Does the behavior change with different compilers (VC2012 vs. VC2013 vs. ICL), or different optimization levels in the same compiler?

(although, yes, I recognize the fact that if it's being miscompiled there's a chance of it being from a legitimate bug that simply exposes it, and the bug needs fixing)

[chainik_svp]

My question is: why is this specific to 64-bit, though?

asked myself the very same question, still no answer :D
may be there's a difference in procedure of loading/unloading DLLs between x32 and x64 modes oO

What I'm getting at is, is there any chance that it's caused by being miscompiled?

not a chance, I can see from the code that some objects are NOT deleted by destructor

===
in fact these bugs are very uncommon
I mean the only chance to face them is to use AVS+ inside ffdshow - then you can load/unload AVS several times in the same process

[chainik_svp]

Just wasted all the Sunday to find out that ScriptEnvironment:: prefetcher is not deleted in destructor.
Which leads to all MT-enabled filter objects are not freed.

[mapg]

Hi,

Let me explain the issue.

Maximum riff index added by ffmpeg in the avi video container is 256, after that point a warning is shown as: "Invalid riff index 257 > 256", later ... "Invalid riff index 258 > 256" and so on ...

Well, when I encode using your Avisynth+ 64-bit and AVISource, when encoder (i.e: x264) reaches riff index 257, the image stays frozen from there to the end of video, namely, scenes stop until the end of video.

Therefore if you are encoding an uncompressed AVI video which is bigger than 256~260 GB (approximately), and the input format is a AVS script (using Avisynth+ and JUST the command: AVISource), at some point of time (when riff index > 256 is reached), the video is frozen, damaged after all.

This problem doesn't happen if I use Avisynth+ and tag DirectShowSource instead of AVISource.

I think that there are two options:

1) Avisynth+ team adds some fix to AVISource to avoid this issue.
2) ffmpeg team allows adding riff indexes bigger than 256 to AVIs.

What do you think?

Let me suggest that you should fix it at your end too.

Kindest regards,
Mapg

BTW: I didn't test if this is also happening in Avisynth+ 32-bit

[colours]

Creating a 257 GB AVI file just to verify this report would be somewhat difficult for the zero active Avisynth+ developers so don't expect any updates on this end.

Besides, this looks more like an FFmpeg issue. FFmpeg would be able to handle over a hundred million RIFF chunks if only one line (http://git.videolan.org/?p=ffmpeg.git;a=blob;f=libavformat/avi.h;h=34da76f715145e976c596459202043b9831ec9e5;hb=HEAD#l32) was changed.

The OpenDML spec doesn't seem to indicate any upper bound on the number of chunks, but I didn't bother reading through it very carefully. It would be pretty weird if there was an artificial 256-chunk limit, though.

[mapg]

As you can see here, there is such limit in ffmpeg (libavformat/avienc.c) ...

if (avi->riff_id > AVI_MASTER_INDEX_SIZE) {
av_log(s, AV_LOG_ERROR, "Invalid riff index %d > %d\n", avi->riff_id, AVI_MASTER_INDEX_SIZE);
return AVERROR(EINVAL);
}Yes, it's certainly weird as none of the most professional video editing software packages add this unfortunate feature.

I will try to forward this issue to ffmpeg team, but I think that AviSynth+ should have a look at this too.

[qyot27]

AVISource relies on the VfW interface to do its parsing (and does so by using VirtualDub's parser, IIRC). If you can open it in VDub but not AviSynth+, then it's an AVISource issue that would need to be resolved by updating AVISource. While FFmpeg might have that limit, it would likely depend on the parser. The DirectShow AVI parser obviously wouldn't have a limit like that if DirectShowSource works. If VDub doesn't like it either, it's probably in VfW and there's little that can be done about it short of finding a third party AVI demuxer for VfW (but honestly, that's probably more trouble than it's worth).

[mapg]

I have tested the issue using FFMS2 and works well, namely there isn't any frozen image once AVI_MASTER_INDEX_SIZE (256GB) is reached.

Definitively the problem is in AVISource from Avisynth+

Mapg

[sl1pkn07]

great news (?)

since wine 1.7.25, can run Avsynth +r1689!

http://sl1pkn07.wtf/bleh56.png

only need in winecfg set msvcr110.dll and msvcp110.dll to 'native,built'

https://bugs.winehq.org/show_bug.cgi?id=17273#c61

greetings

[qyot27]

Also confirmed working with VC2013 builds (msvcp120.dll and msvcr120.dll).

But getting it working in Wine 1.7 requires the workaround I posted earlier (http://forum.doom9.org/showpost.php?p=1663971&postcount=525). Trying to install the 2012 and 2013 runtimes from vcredist_*.exe fails, requiring the user to make sure they get the DLLs themselves and install them (read: copy into system32). The built-in version errors out on the GetConcurrency function, but at least it can actually use the Microsoft runtimes now and operate.

[sl1pkn07]

I tried the 64bits version (with 'win64' winearch) and don't work. tested with 'Windows XP' and 'windows 7' mode

the 32bits only work in 'Windows XP' mode (see the capture avobe). but fail in 'Windows 7' mode

for installation (include vcredis 2012), first need set the wine with 'windows 7' mode, install avs+ r1689 from line0, then back to 'Windows XP' mode and set msvc{r,p}110.dll to native with winecfg

tested with 3 clean prefixes, wine only 32bits (winearch win32), wine only 64bits (winearch win64) and wine mixed (clear winearch variable)

the message is: failed to load avisynth.dll

greetings

[martin53]

I know similar questions have been asked in this forum more than once, but that was before AviSynth+, which released the tight doctrine a bit, and maybe less specific.

Is it possible in general (and worthwile...) that the gurus add a Dictionary element to the VideoInfo interface, together with methods to add and delete a key/item pair, and to get the item for a specific key, and also make the latter methods available to scripts?

Specifically, a pair of functions could then exchange information through this interface.

Say
function 1 prepares a 3d- or 16-bit, or certain colorspace frame so that a standard filter chain, which is older and not 100% compatible, can handle this clip. E.g. 3d side-by-side video could be cropped and interleaved, then processed as double frame rate 2d clip.
Function 2 stands at the end of the filter chain, and will automatically reverse the "tunneling", if it knows that there is need to. Some plugins used "hints" for that, i.e. manipulated certain pixel LSBs, etc; but this is not robust against filtering.

In my imagination, function 1 could call Add(clip, key, string) to add the dictionary entry (key,string) to the clip's VidoInfo, and this would be forwarded through the filter chain like the other clip properties.
Function 2 could call ExistsKey(clip, key), and/or GetKey(clip, key), which would signal, resp. return the string.
This could look like

#Assume Function1 calls MediaInfo to get information about the clip display ratio, which is not width:height ratio, i.e. nonsquare pixels
#Function1 gives the clip this freename property
AddKey("DisplayRatio", string(retrieved_display_ratio))

... #standard filter chain with unchanged width/height
... #deblocking and other filters need unstretched, uncropped input

#function2
GetKey(clip, "DisplayRatio")
\ ? BiCubicResize(...)
\ : last
#as the last filter in the chain, function2 sets the output width/height ratio in order to meet display ratio with square pixels
...
One more example to make it as clear as possible: There is a function 'ConvertBackToYUY2()' which should be used when the clip originally was converted from YUY2 to a different color space. It would be more elegant to attach the property that it was originally YUY2 to a clip - compared to today's script language limits.

EDIT: here (http://forum.doom9.org/showthread.php?p=1709612#post1709612) is another example for the usefulness of arbitrary clip properties ('tunneled' in this case)

[qyot27]

Time for a test build, I guess:

AviSynth+ r1718 (http://www.mediafire.com/download/f3egmdgdkwy8dmf/avisynthplus_r1718-20150301.7z)

Basically, I merged in most of the 22 commits that make up 2.6 RC1. Some are mostly applied 1:1 (including the 7 HTML docs commits), others are adapted (a couple of them heavily cut down from their original form). Four of the commits were outright rejected because they're irrelevant for AviSynth+.

This is also probably a better option than continuing to rely on r1689, since there were some MT fixes applied after r1689.


Built with Visual Studio 2013, and it includes both 32-bit and 64-bit .dlls (no 64-bit DirectShowSource because I didn't feel like trying to get the SDKs lined up first). Note that I do not have access to a 64-bit Windows install, so those of you that do, please confirm that the 64-bit .dlls work.

[burfadel]

Thanks, been looking forward to an update!

However, the download doesn't work. It tries to download then fails (tried different browsers). Any chance of another mirror?

EDIT:
Kept trying, then Mediafire showed the 'repair download' screen. After that it worked.

[ultim]

All changes from 2.6rc1 have been pulled into AviSynth+. Normally I would publish a new build, but qyot has already done so and there are only very insiginifcant differences to his build. I will now look at the memory leak reported by chainik and the audio issue. At that point Avs+ will be good to go for a new official release that is stable in single-threaded mode. I still wouldn't recommend the MT-modes for production, but everything else should be alright. And of course, anything else you find, report it, so that it can be fixed timely. Sorry for the hiatus.

[nixo]

Excellent news, thank you.

@qyot27
The 64-bit .dlls work fine for me. Thanks!

[Sparktank]

This is intriguing news, indeed!

Great work, guys.

[aegisofrime]

All changes from 2.6rc1 have been pulled into AviSynth+. Normally I would publish a new build, but qyot has already done so and there are only very insiginifcant differences to his build. I will now look at the memory leak reported by chainik and the audio issue. At that point Avs+ will be good to go for a new official release that is stable in single-threaded mode. I still wouldn't recommend the MT-modes for production, but everything else should be alright. And of course, anything else you find, report it, so that it can be fixed timely. Sorry for the hiatus.

Great news, good to see you back! I have been holding off on switching to Avisynth 2.6 because of how well Avs+ works for me!

[burfadel]

Do the crashes only affect when using MT? I don't use that. I have been using the 1718 build without issue. This includes a backlog of stuff I wanted to encode, so encoding dozens of shows (with very 'tight' x264 settings etc) over the last couple of days, non-stop. Haven't had any issues :).

[jpsdr]

At that point Avs+ will be good to go for a new official release that is stable in single-threaded mode.
Thanks, and it's nice to see you back.

[ryrynz]

I just wanna see something that can actually replace SEt's MT builds :P
Welcome back to the land of development.

[ultim]

I just wanna see something that can actually replace SEt's MT builds :P
Welcome back to the land of development.

One of the largest issues with Avs+ in MT mode is discussed in issue 37 (http://github.com/AviSynth/AviSynthPlus/issues/37). It also affects more filters than just those mentioned in the issue, many of them relatively popular.

[ultim]

[for every frame]
{
VideoFrame *frame = it->second;
if (frame->vfb->refcount == 0)
delete frame->vfb;

delete frame;
}

why "if (frame->vfb->refcount == 0)" ??
vfb->refcount decremented in VideoFrame destructor so it can be (and it IS) >0
and ~VideoFrame doesn't delete vfb but just releases it

=====
Fixed it, see https://github.com/AviSynth/AviSynthPlus/issues/49

Is there any chance to include this fix to the main branch and release a new official build?

It's not that simple. As you see, in that loop all frames get deleted. Each such deletion of a VideoFrame will decrease the refcount on the corresponding vfb. Since vfb's are only referenced by VideoFrames, when the last VideoFrame referencing that vfb gets deleted, the condition with "(frame->vfb->refcount == 0)" will get triggered and the vfb will get deleted too. Hence after enough VideoFrames have been deleted, eventually the vfb will too, even if the "zero-condition" wasn't true at first.

The proposed solution in the issue tracker won't work well. You cannot delete the vfb every time a VideoFrame is destroyed, because due to subframes, vfbs can be shared among multiple VideoFrames. Of course you could check for the zero-refcount-condition and only delete it when no one else is using it, but that is basically what the ScriptEnvironment's destructor is already doing, except your proposed solution would destroy caching the vfbs, negatively affecting performance.

EDIT: You might probably be thinking, because the refcount is decremented in VideoFrame's destructor, if I want to check for the last reference, I should be checking for refcount==1 instead of refcount==0. The reason this is not the cases, is because all explicit references to VideoFrames will be released along the destruction of the filter chain. Because releasing a videoframe will also decrement the associated vfb's refcount, vfb refcounts will decrement to zero (except for those used by subframes) even before the videoframes are destructed. So to sum it up, the error that you are making is to assume that vfb::Release is only called in VideoFrame's destructor.

[ultim]

this isn't all the truth :D

if the filter was instantiated - it'll never be deleted...

in other words if you'll write some filter doing something like this:

SomeFilter::SomeFilter()
{ buffer = new BYTE[1000000000]; }

SomeFilter::~SomeFilter()
{ delete[] buffer; }

than you'll end with 1 GB of memory stolen cause it'll never be freed :)

Not quite true. Filters are only referenced by each other, except for the top-level clip, which is referenced by the hosting application. Hence when the host application releases his own filter reference (PClip), that will trigger the destruction of the whole filter chain (when a filter's reference reaches zero, it gets deleted as per IClip::Release). In the end every filter will be destroyed.

This is also true about the other supposed "problem" you have found about not destroying the Prefetcher. The Prefetcher is just a filter, so it also get destroyed during the destruction of the filter chain, as explained above. Sure, ScriptEnv also has a reference to it, but it never incremented its refcount, so this extra reference won't prevent its timely destruction when the host app releases the chain.

To sum it, I see no memory leak yet in any of the three cases you have documented so far. Please elaborate more if you think I have missed something.

[chainik_svp]

ultim
I believe theory is great :)

But how many times in real life you actually checked what is going on at script destruction?
I know it's not a usual case. Most people will never face any problems related to script destruction.

BUT it's important for real-time processing inside ffdshow when AVS script can be reloaded many times while watching the same movie.
It isn't very hard to make SVP "compatible" with AVS+. If you're interested I can share a few files (actually it's just one file) to make it work.
As an alternative you can just write any simplest script inside ffdshow and check and uncheck "avisynth" checkbox.

And the truth is every script reloading with AVS+ makes visible step in the Task manager's memory usage graph.
So the memory _is_ leaking. And after my fix it is not.

I'm not telling the fix is 100% correct, but at least it can point you to the right direction.

But if you insist I can point you to exact lines of code that break your theory ;)

In fact there's no point for arguing, just try it yourself :)

[ultim]

ultim
I believe theory is great :)

But how many times in real life you actually checked what is going on at script destruction?
I know it's not a usual case. Most people will never face any problems related to script destruction.

BUT it's important for real-time processing inside ffdshow when AVS script can be reloaded many times while watching the same movie.
It isn't very hard to make SVP "compatible" with AVS+. If you're interested I can share a few files (actually it's just one file) to make it work.
As an alternative you can just write any simplest script inside ffdshow and check and uncheck "avisynth" checkbox.

And the truth is every script reloading with AVS+ makes visible step in the Task manager's memory usage graph.
So the memory _is_ leaking. And after my fix it is not.

I'm not telling the fix is 100% correct, but at least it can point you to the right direction.

But if you insist I can point you to exact lines of code that break your theory ;)

In fact there's no point for arguing, just try it yourself :)

I think you misunderstand me. I'm not saying there is no memory leak. In fact, I have also seen memory leaking myself. I'm just saying it has other reasons than the ones you explained. I am still trying to pinpoint the exact reason though.

But how many times in real life you actually checked what is going on at script destruction?
LOL, you think all I do is make up theories? Believe me when I say I checked it in multiple scenarios. I know something is leaking, I was just saying the problem is not what you think it is.

[chainik_svp]

ultim
I was just saying the problem is not what you think it is

Yeah, ok, whatever :D

Last time I was heard this - I was forced to go deep into Avisynth code and make my own build, with completely replaced 'distributor' and modified cache.
And I really don't want to go into this [shit] one more time.

If you can make AVS+ useful - that would be great, cause I really like how it works with the memory and with CPU threads in MT version.

But now isn't useful at all :(

[chainik_svp]

To sum it, I see no memory leak yet in any of the three cases you have documented so far. Please elaborate more if you think I have missed something.

Is it enough to put additional printf in my filters's destructor and see that it was never called?
Or may be put that printf into ~Prefetcher() ?

[ultim]

But if you insist I can point you to exact lines of code that break your theory ;)

The .avs script that is leaking memory and a part of the source video (some frames of it but with same compression and video format) should help a lot to reproduce. With simple scripts I tested, there are no leaks ATM, so I will probably need to use the same filters as you do to see it.

[jmac698]

- Script language extensions, with support for multiline conditionals and loops.
This is something I've always wondered about. Where is it documented? Can you give an example of these new script commands? Is it something like gscript?

Also, is there any need to use official avisynth now? If not, have the two authors considered a merge?

[ryrynz]

One of the largest issues with Avs+ in MT mode is discussed in issue 37 (http://github.com/AviSynth/AviSynthPlus/issues/37). It also affects more filters than just those mentioned in the issue, many of them relatively popular.

What are the options? Not professing to know anything about the issue but can you just do whatever AvisynthMT does? Or use a compatibility list and address specific requirements for the filters based on that?

My thoughts are as long as the solution provides at least the same performance as AvisynthMT it's a win regardless of how hacky it may seem.

[martin53]

latest release: 2014. Jan. 02.
avisynth+ features for users
[…]
- support for multiple ("shadow") plugin directories
- autoloading of c-plugins
- script language extensions, with support for multiline conditionals and loops.
- improved still image support and timestretch
[…]
compatiblity to avisynth
avisynth+ tries to provide a superset of avisynth's features while staying compatible to existing code […]


I feel sorry to ask this question here, since from quick reading the thread, it becomes clear that there was (and hopefully is) put much effort in this creditable modernization.

That said, my question is: where is the documentation of the cited new features? I may be too stupid to find it. After Avisynth+ install, I compared the doc folder trees with BeyondCompare and can only find same docs, or files where AviSynth 2.6 is more up to date (even if Avisynth+ file date is newer!). The folder 'german' should be renamed 'french', or better removed, if it is neither complete nor updated. Also, I found nothing on the homepage (http://avs-plus.net/) and at most hints in this thread.

EDIT: reading my own text, I feel I should maybe offer to contribute. Would that be welcome, even at a low level, and which prerequisites do I need (e.g. I din't have experience with git)?

[martin53]

MT-enabled AviSynth+ triggers a latent bug in AvsPmod. Until a new version of AvsPmod is officially released, use this (http://files.line0.in/builds/AvsPmod-2.5.1-r426-x86-04874ed.7zl) build. A thousand thanks to vdcrim for the fix.
Unfortunately, the file does no longer exist. Can it please be published again?

EDIT 2015-03-09: link updated in original post, and also here

[martin53]

With both the r1576 and the r1718 build as x86 under a X64 windows 7, this function lister script (http://forum.doom9.org/showthread.php?p=1712503#post1712503) freezes after debug output 'ImageSeq_ImageWriter', even if call to all ImageSeq_ functions is avoided. Maybe the following function doesn't even like to be checked by RT_PluginFunctions().
With the given set of pre-captured (i.e. not called) functions (line #58 ff), the same script runs fine with Avisynth 2.6 ST and MT.

Supplement: Avisynth 2.6 MT shows GetMTMode and SetMTMode as internal functions. With Avisynth+ I don't see them in the list.

[ultim]

This is something I've always wondered about. Where is it documented? Can you give an example of these new script commands? Is it something like gscript?

It is mostly like GScript. In fact, GScript has been used as the starting point for the implementation in Avs+, and changes compared to GScript are not visible to the user (e.g. only important for Avs+ core developers). Two notable differences are between GScript and Avs+:

In Avs+ there is no need to use GScript(""" and """) to encompass your GScript-specific code parts. The language extensions became native to Avs+ and can be used transparently like classic Avisynth syntax.
The "return" statement has been slightly changed to not only exit the inner-most code block, but to terminate the whole function (or script), as anybody with even the slightest scripting experience would expect. This is one of the very few incompatible changes compared to classic Avisynth.


Also, is there any need to use official avisynth now?
Even though the many changes, Avisynth+ is still highly compatible to classic Avisynth, with respect to both scripts and plugins - with only very few exceptions. Unless you find a bug in Avisynth+ that isn't there in Ianb's version, you should be able to use Avs+ as a complete replacement for Avs. If you do find a bug in Avs+ that is unique to it, I'd be happy if you reported it so that we can fix it, instead of just silently switching back to the old Avs.

If not, have the two authors considered a merge?
A proposed merge was the first thing Avs+ has started out as. Only after it failed did Avs+ become a separate project. Communication with "upstream" went south early in the beginning, and I am to blame too. Nevertheless, upstream has been unresponsive since then even while we were trying to contribute, so I guess a full-blown merge is pretty much out of the question as things are right now.

[ultim]

What are the options? Not professing to know anything about the issue but can you just do whatever AvisynthMT does? Or use a compatibility list and address specific requirements for the filters based on that?

My thoughts are as long as the solution provides at least the same performance as AvisynthMT it's a win regardless of how hacky it may seem.

Various options are being considered of course, each with their own advantages. MT is not forgotten, but I do admit I am currently focusing on the single-threaded scenario for the next release. MT will have to wait until then. Due to all the historic baggage Avisynth has, it is very hard to introduce invasive changes like MT correctly without breaking old stuff.

[ultim]

I feel sorry to ask this question here, since from quick reading the thread, it becomes clear that there was (and hopefully is) put much effort in this creditable modernization.

That said, my question is: where is the documentation of the cited new features? I may be too stupid to find it. After Avisynth+ install, I compared the doc folder trees with BeyondCompare and can only find same docs, or files where AviSynth 2.6 is more up to date (even if Avisynth+ file date is newer!). The folder 'german' should be renamed 'french', or better removed, if it is neither complete nor updated. Also, I found nothing on the homepage (http://avs-plus.net/) and at most hints in this thread.

Nah, it's really not your fault. There is no centralized documentation so far. All features have been documented in this thread as they were released (some from the start might also be in the classic Avisynth thread), but people are - very unfortunately - left to find these posts on their own. So it is not your fault. At one point we will have to collect all these scraps of information, make proper html of it, and ship it as part of the documentation in the installer. That would be ideal, but nobody yet took the time for it. Me too, I'd rather code, fix bugs and introduce new features, than spend hours on documenting - bad habit for most programmers.

This is an area where non-programmers among you could help out (writing docs), and these contributions too would be more than welcome.

[martin53]

[...]documentation[...]is an area where non-programmers among you could help out (writing docs), and these contributions too would be more than welcome.

Your response overlapped with my edit. Could you please post a guide what to install and how to merge the edits - probably with git, like the sources itself?

[ultim]

Unfortunately, the file does no longer exist. Can it please be published again?

I now updated the link in the original post. Please try again and thanks for letting me know.

[chainik_svp]

ultim
The .avs script that is leaking memory and a part of the source video (some frames of it but with same compression and video format) should help a lot to reproduce. With simple scripts I tested, there are no leaks ATM, so I will probably need to use the same filters as you do to see it.

You won't see those leaks while running any script.
http://forum.doom9.org/showthread.php?p=1697682#post1697682
enter simple script into ffdshow:
SetFilterMTMode("",2)
SetFilterMTMode("ffdShow_source",3)
ffdShow_source()
BilinearResize(3000,1500)
BilinearResize(1920,1080)
Prefetch(10)


and then just check and uncheck "Avisynth" check box while looking at Task Manager memory graph

[ultim]

With both the r1576 and the r1718 build as x86 under a X64 windows 7, this function lister script (http://forum.doom9.org/showthread.php?p=1712503#post1712503) freezes after debug output 'ImageSeq_ImageWriter', even if call to all ImageSeq_ functions is avoided. Maybe the following function doesn't even like to be checked by RT_PluginFunctions().
With the given set of pre-captured (i.e. not called) functions (line #58 ff), the same script runs fine with Avisynth 2.6 ST and MT.

Created ticket, and will look at it soon:
https://github.com/AviSynth/AviSynthPlus/issues/55

Supplement: Avisynth 2.6 MT shows GetMTMode and SetMTMode as internal functions. With Avisynth+ I don't see them in the list.
That is normal. GetMTMode() ans SetMTMode() are not part of Avisynth+, as Avisynth+ uses other functions to achieve multithreading.

[ultim]

Your response overlapped with my edit. Could you please post a guide what to install and how to merge the edits - probably with git, like the sources itself?

I will try to today, give me some time.

[jmac698]

Ultim,
Thanks for your response. It is good news. I remember I hesitated at first to use GScript because it required an extra plugin, but I thought it should be added to Avisynth anyhow. Now my wish has come true :)

[qyot27]

After Avisynth+ install, I compared the doc folder trees with BeyondCompare and can only find same docs, or files where AviSynth 2.6 is more up to date (even if Avisynth+ file date is newer!). The folder 'german' should be renamed 'french', or better removed, if it is neither complete nor updated. Also, I found nothing on the homepage (http://avs-plus.net/) and at most hints in this thread.

EDIT: reading my own text, I feel I should maybe offer to contribute. Would that be welcome, even at a low level, and which prerequisites do I need (e.g. I din't have experience with git)?
That would be an installer issue if the French docs are being put in the German folder. In the source tree, the German docs are in the German folder.

There shouldn't be any cases where 2.6's docs are more up-to-date than avsplus'. If you're comparing r1576 to 2.6 RC1, then there's an obvious reason: r1576 is from December 2013. The docs changes that were part of RC1 (7/22 of the commits that comprise the difference between Alpha 5 and RC1) were merged to avsplus' Git repository with the rest of the RC1 integration about a week ago.

[ultim]

With both the r1576 and the r1718 build as x86 under a X64 windows 7, this function lister script (http://forum.doom9.org/showthread.php?p=1712503#post1712503) freezes after debug output 'ImageSeq_ImageWriter', even if call to all ImageSeq_ functions is avoided. Maybe the following function doesn't even like to be checked by RT_PluginFunctions().
With the given set of pre-captured (i.e. not called) functions (line #58 ff), the same script runs fine with Avisynth 2.6 ST and MT.

I can run the script to the end just fine, both with GScript as well with the native language syntax in Avs+. For reference, here is my output (http://pastebin.com/UVVH9zx6). So I guess it really has to do something with that specific plugin that "doesn't like to be checked", at least not in combination with Avs+. Can you figure out which plugin it is? Then send me the plugin please so that I can investigate further.

Edit: As a side note, your script doesn't work well perfectly if some functions have the same name but different argument list. In that case it will always list the same variant. That is the same though under classic Avisynth too, not specific for Avs+ .

[ultim]

That would be an installer issue if the French docs are being put in the German folder. In the source tree, the German docs are in the German folder.

Actually, I've been meaning to remove non-english docs. They are out of date anyway, and it is unlikely they will ever have the same kind of maintenance as the english docs.

[qyot27]

Actually, I've been meaning to remove non-english docs. They are out of date anyway, and it is unlikely they will ever have the same kind of maintenance as the english docs.
That works too. Users can always consult the avisynth.ru docs mirror, or the Wiki on avisynth.nl (which would also be a reason to slice the externalfilters/ stuff out of the English docs, or at least spin the RST versions of them off into a separate repository).

[martin53]

I can run the script to the end just fine [...] Can you figure out which plugin it is? [...]
I'll gladly do, just working on it.

Edit: As a side note, your script doesn't work well perfectly if some functions have the same name but different argument list. In that case it will always list the same variant. That is the same though under classic Avisynth too, not specific for Avs+ .

Interesting. I always wondered why identical function_parameter lines appeared more than once. If Groucho2004 extended the AVSInfoTool to display the parameter lists, we'd probably know if it's a bug in AviSynth or in RT_Stats. Actually, I re-included a check that prevents similar lines also for internal functions, not only for plugins (updated my post in RT_Stats thread) though I understand it would be better to list the functions with their multiple parameter lists.

[martin53]

Can you figure out which plugin it is?
The behavior appears to be non reproducible. List diff here (http://www.mediafire.com/download/scsyp3fhbbc3fio/AllFunctionsDiffReport.html.zip).
I installed Avisynth+ r1576 again (previously had done an uninstall that reactivated 2.6 MT) and ran the script with freeze once in AvsPmod. Then looked into the script to could add another RT_DebugF() output and then noticed that's nonsense because all functions are retrieved at once into a string and the current debug output is just in the right place. After that, I ran the script successfully several times in AvsPmod and once in VDub portable. But freeze after "F2" in VDub. Next time again freeze in AvsPmod, when defreq_DeFreq was called. This was also the freezing plugin in the first run.
An explanation why it differs from my ealier post is that I then listed the plugins in reverse order (so the line 'plugin_function' comes first and I can reject the similar line 'function' without 'plugin_' prefix). Meanwhile I changed the rejection method and can also reject 'function' if it comes before 'plugin_function'.

EDIT: Reproduced freeze during DeFreq (http://avisynth.org.ru/defreq/defreq.html)call. %PATH% includes AviSynth plugins directory (which is actually a NTFS symbolic link to another partition so I include it in the backup of that partition) and Defreq-required FFTW3.dll with date 06-21-2010 and 1.46MB is there.
But after DeFreq is removed, AvsPmod freezes in DePanEstimate.
I get more and more the feeling that it is a string/memory issue, not a plugin issue. Changed the script accordingly to following code, now always freezes. As it seems, all my FFT plugins trigger it. I remove one, the next one triggers the freeze. My suspicion is they allocate much memory, but why is that not either succesful or fails with an error, but instead enters an endless loop?
Try {RT_Version()} catch (msg) {assert(false,"This script requires the RT_Stats plugin.")}
Try {GEval("")} catch (msg) {assert(false,"This script requires the GScript plugin.")}

#============================================================================
Filename=RT_FSelSaveAs(title="Save Filter list",dir="",filt="Text files|*.txt",fn="_AllFunctions.txt",debug=true)
Assert(FileName.IsString(), "Please provide a file name to save the filter list")

#============================================================================
RT_WriteFile(Filename, VersionString(), Append=false)
#WriteInternalFunctionsSyntax(RT_InternalFunctions(), Filename)
WritePluginFunctionsSyntax(RT_PluginFunctions(), Filename)
WritePluginFunctionsSyntaxBackward(RT_PluginFunctions(), Filename)
WritePluginFunctionsSyntax(RT_PluginFunctions(), Filename)
WritePluginFunctionsSyntaxBackward(RT_PluginFunctions(), Filename)
BlankClip().subtitle("Function info is complete.",align=5)

#============================================================================
function WriteInternalFunctionsSyntax(string fu_list, string Filename) {
GScript("""
done=""
while(fu_list != "") {
i = FindStr(fu_list+" ", " ")
fu_name = LeftStr(fu_list, i-1)
fu_list = MidStr(fu_list, i+1)
params = RT_PluginParam(fu_name)
if (FindStr(done, fu_name+params)==0) {
done = done + " "+fu_name+params
RT_WriteFile(Filename, "internal %s ( %s ) -> %s", fu_name, params, GetReturnType(fu_name, params), Append=true)
}
}
""")
}
#============================================================================
function WritePluginFunctionsSyntax(string fu_list, string Filename) {
GScript("""
done=""
while(fu_list != "") {
i = FindStr(fu_list+" ", " ")
fu_name = LeftStr(fu_list, i-1)
fu_list = MidStr(fu_list, i+1)
i = FindStr(fu_name, "_")
if (i>0) {
params = RT_PluginParam(fu_name)
if (FindStr(fu_list, "_"+fu_name)==0 && FindStr(done, "_"+fu_name+params)==0) {
done = done + " _"+fu_name+params
RT_WriteFile(Filename, "plugin %s ( %s ) -> %s", fu_name, params, GetReturnType(fu_name, params), Append=true)
}
}
}
""")
}
#============================================================================
function WritePluginFunctionsSyntaxBackward(string fu_list, string Filename) {
GScript("""
done=""
fu_list = RevStr(fu_list)
while(fu_list != "") {
i = FindStr(fu_list+" ", " ")
fu_name = RevStr(LeftStr(fu_list, i-1))
fu_list = MidStr(fu_list, i+1)
i = FindStr(fu_name, "_")
if (i>0) {
params = RT_PluginParam(fu_name)
if (FindStr(fu_list, "_"+fu_name)==0 && FindStr(done, "_"+fu_name+params)==0) {
done = done + " _"+fu_name+params
RT_WriteFile(Filename, "plugin %s ( %s ) -> %s", fu_name, params, GetReturnType(fu_name, params), Append=true)
}
}
}
""")
}
#============================================================================
#function GetReturnType(string fu, string params) { return "?" }
#============================================================================
function GetReturnType(string fu, string params) {
GScript("""
RT_DebugF("Fu=%s %s", fu, params, name="GetReturnType")
if (MidStr(fu,3,1)=="+") {fu = LeftStr(fu,2)+MidStr(fu,2)}
if (MidStr(fu,2,1)=="+") {fu = LeftStr(fu,1)+fu}
else if (fu=="Assert") {return "conditional halt"}
else if (FindStr(fu, "avss_26_DSS2")>0) {return "clip"}
else if (FindStr(fu, "ImageSeq_")>0) {return "clip"}
else if (FindStr(fu, "LoadCPlugin")>0) {return "void"}
else if (FindStr(fu, "LoadVirtualdubPlugin")>0) {return "void"}
else if (FindStr(fu, "MDepan")>0) {return "clip"}
else if (FindStr(fu, "MSCDetection")>0) {return "clip"}
else if (FindStr(fu, "ResampleAudio")>0) {return "clip"}
else if (FindStr(fu, "Resize")>0) {return "clip"}
else if (FindStr(fu, "RT_Debug")>0) {return "clip"}
else if (FindStr(fu, "RT_FSel")>0) {return "."}
else if (FindStr(fu, "Source")>0) {return "clip"}
else {
bc = BlankClip(width=8,height=8,length=1,pixel_type="YV12")
fp = LeftStr(params,2)=="cc" ? "(bc,bc)"
\ : LeftStr(params,2)=="cf" ? "(bc,0.1)"
\ : LeftStr(params,3)=="cci" ? "(bc,bc,0)"
\ : LeftStr(params,5)=="ciiii" ? "(bc,0,0,0,0)"
\ : LeftStr(params,4)=="ciii" ? "(bc,0,0,0)"
\ : LeftStr(params,3)=="cii" ? "(bc,0,0)"
\ : LeftStr(params,2)=="ci" ? "(bc,0)"
\ : LeftStr(params,1)=="c" ? "(bc)"
\ : LeftStr(params,2)=="ff" ? "(0.1,0.1)"
\ : LeftStr(params,1)=="f" ? "(0.1)"
\ : LeftStr(params,2)=="ii" ? "(0,0)"
\ : LeftStr(params,1)=="i" ? "(0)"
\ : LeftStr(params,2)=="ss" ? "("+chr(34)+"1"+chr(34)+","+chr(34)+"1"+chr(34)+")"
\ : LeftStr(params,2)=="si" ? "("+chr(34)+"1"+chr(34)+",1)"
\ : LeftStr(params,1)=="s" ? "("+chr(34)+"1"+chr(34)+")"
\ : "()"
Try {
r = Eval("current_frame=0 "+fu+fp)
r = r.IsBool ? "bool" : r.IsClip ? "clip" : r.IsInt ? "int" : r.IsFloat ? "float" : r.IsString ? "string" : "?"}
catch (msg) { Try {r = r.IsBool ? "bool" : r.IsClip ? "clip" : r.IsInt ? "int" : r.IsFloat ? "float" : r.IsString ? "string" : "?" } catch(msg) {r = "?"}
}
if (r=="?") {
bc = BlankClip(width=8,height=8,length=1,pixel_type="RGB32")
fp = LeftStr(params,2)=="cc" ? "(bc,bc)"
\ : LeftStr(params,2)=="cf" ? "(bc,0.1)"
\ : LeftStr(params,3)=="cci" ? "(bc,bc,0)"
\ : LeftStr(params,5)=="ciiii" ? "(bc,0,0,-0,-0)"
\ : LeftStr(params,4)=="ciii" ? "(bc,0,0,0)"
\ : LeftStr(params,3)=="cii" ? "(bc,0,0)"
\ : LeftStr(params,2)=="ci" ? "(bc,0)"
\ : "(bc)"
Try {
r = Eval("current_frame=0 "+fu+fp)
r = r.IsBool ? "bool" : r.IsClip ? "clip" : r.IsInt ? "int " : r.IsFloat ? "float" : r.IsString ? "string" : "?"
} catch (msg) {
r = "?"
}
}
if (r=="?" && LeftStr(params,1)=="c") {r="clip?"}
return r
}
""")
}

[Wilbert]

or the Wiki on avisynth.nl (which would also be a reason to slice the externalfilters/ stuff out of the English docs
Will move them out.

[ultim]

Your response overlapped with my edit. Could you please post a guide what to install and how to merge the edits - probably with git, like the sources itself?

Exactly, with Git, just like you would handle the source itself. Actually, Git doesn't (and cannot really) differentiate between sources and docs at all.

First, you'll need a GitHub account. Then get GitHub for Windows (https://windows.github.com/), which will be your offline GUI to Git. Then you just follow this (https://guides.github.com/activities/forking/) process while editing the docs with your text editor/html-viewer. If you will be working on the docs and not on the sources, there aren't any more tools you'll need.

[ultim]

Changed the script accordingly to following code, now always freezes.

I checked the new script. True this one "freezes", more exactly it keeps running forever by never stopping to output. It outputs all functions A-Z, then backwards Z-A, then again A-Z, then Z-A, and seems to keep on doing this forever. Judging by this regular output pattern, are you sure it is not a bug in your script? See output here (http://pastebin.com/6Svj5TnL).

Edit: After letting it run loooong enough, it actually finished execution! Is it normal to output everyhting A-Z-A-Z-A a couple of time?

[ultim]

You won't see those leaks while running any script.
http://forum.doom9.org/showthread.php?p=1697682#post1697682

I tried it now with ffdshow (http://sourceforge.net/projects/ffdshow-tryout/?source=typ_redirect)(in MPC-HC) and your script. Only difference is I used 5 instead of 10 threads as I have a 4-core CPU. Toggled the AviSynth-option in ffdshow on and off at least a dozen of times, and memory consumption was always about 330Mbytes in task manager when switched on. Said differently, I see no obvious memory leak.

I looked at the date when you originally reported this, and it was in oct.2014. Given that the last release is from 2013, I think this bug might have been corrected since then. There has been a LOT of fixes since that release, and it would explain why you see this bug and I don't. Anyway, I'll publish a new test binary of Avs+ in a couple of days, and let me know if the problem still pertains.

[Octo-puss]

Oh the development is back. Very nice!

[chainik_svp]

utim
I looked at the date when you originally reported this, and it was in oct.2014

I was building latest git sources back in October. In fact I was interested in x64 MT version at the first place so almost all my tests was about it.
And I'm now looking at those two builds I've made and yeah, x32 and x64 versions are working in different ways.

For 720p video and 10 MT threads: x32 build stays at about 500 MB, and x64 adds +350 MB on each click on the checkbox.

[ultim]

utim
I looked at the date when you originally reported this, and it was in oct.2014

I was building latest git sources back in October. In fact I was interested in x64 MT version at the first place so almost all my tests was about it.
And I'm now looking at those two builds I've made and yeah, x32 and x64 versions are working in different ways.

For 720p video and 10 MT threads: x32 build stays at about 500 MB, and x64 adds +350 MB on each click on the checkbox.
Ok, my bad. I did my experiemnt with ffdshow on the i386 build. I'll have to repeat it for x64. and I will let you know of the results.

[Groucho2004]

For 720p video and 10 MT threads: x32 build stays at about 500 MB, and x64 adds +350 MB on each click on the checkbox.
I have not tried ffdshow but loading the IScriptEnvironment, reading some frames, releasing with "DeleteScriptEnvironment()" several consecutive times does not increase the memory usage with the latest AVS+ x64 build.

[martin53]

Two notable differences are between GScript and Avs+ (http://forum.doom9.org/showthread.php?p=1648026#post1648026):

In Avs+ there is no need to use GScript("...") to encompass GScript-specific code parts.
The "return" statement terminates the whole function (or script).


Third, with original GScript, I think the variable 'last' is not fed through the GScript part, so scripts need to assign all results to explicit variables (maybe it's only roughly like that, I remember I had to fix some scripts bugs of this kind).

Has this been changed in Avs+, too, or is it still same?

[Reel.Deel]

my question is: where is the documentation of the cited new features? I may be too stupid to find it. After Avisynth+ install, I compared the doc folder trees with BeyondCompare and can only find same docs, or files where AviSynth 2.6 is more up to date (even if Avisynth+ file date is newer!). The folder 'german' should be renamed 'french', or better removed, if it is neither complete nor updated. Also, I found nothing on the homepage (http://avs-plus.net/) and at most hints in this thread.


This morning I started the AviSynth+ wiki page (http://avisynth.nl/index.php/AviSynth%2B) in hopes to document all of the new features, changes, bugs, etc, really anything that can be helpful to end users and programmers looking to contribute to AviSynth+. Right now the page is sloppy but I'm hoping to improve it in the coming days. Anything that you would like to add/clarify just let me know or edit the wiki yourself :).

[martin53]

i checked the new script. True this one "freezes", more exactly it keeps running forever by never stopping to output. It outputs all functions a-z, then backwards z-a, then again a-z, then z-a, and seems to keep on doing this forever. Judging by this regular output pattern, are you sure it is not a bug in your script? see output here (http://pastebin.com/6svj5tnl).

Edit: After letting it run loooong enough, it actually finished execution! Is it normal to output everyhting a-z-a-z-a a couple of time?

Yes, exactly this was the change to make it 'freeze' more likely.

Saw it freezing once today with 2.6 MT, but can't reproduce. But it has to do with string handling or so, not at all with plugins. I learned that there's no garbage collection in Avisynth, but it looks very similar to a veeerrry slooow one, or desperate tries to allocate memory. It tended to 'freeze' just after the long string containing all function names was emptied by the while loop, though that was not the only situation.

Will report further success.

[ultim]

utim
I looked at the date when you originally reported this, and it was in oct.2014

I was building latest git sources back in October. In fact I was interested in x64 MT version at the first place so almost all my tests was about it.
And I'm now looking at those two builds I've made and yeah, x32 and x64 versions are working in different ways.

For 720p video and 10 MT threads: x32 build stays at about 500 MB, and x64 adds +350 MB on each click on the checkbox.

Okay, I managed to reproduce it under x64 ffdshow. I am inclined to say however, that since the leak only occurs under ffdshow and only in x64, that ffdshow should be checked first. No leak in x64 avisynth with either avsmeter, virtualdub, or mpc-hc, and also no leak in 32-bit ffdshow. It only happens with x64 ffdshow. Can someone please check the ffdshow sources?

[ultim]

Yes, exactly this was the change to make it 'freeze' more likely.

Saw it freezing once today with 2.6 MT, but can't reproduce. But it has to do with string handling or so, not at all with plugins. I learned that there's no garbage collection in Avisynth, but it looks very similar to a veeerrry slooow one, or desperate tries to allocate memory. It tended to 'freeze' just after the long string containing all function names was emptied by the while loop, though that was not the only situation.

Will report further success.

Oh my gosh, I bet it has to do something with AviSynth's StringDump. This is only a hunch though. Is it reproducible on classic Avisynth, even if only sporadically?

[chainik_svp]

ultim

The only question left is how I was able to remove memory leaks with those two patches to AVS+? :)

Just try to believe that cache is not cleaned and prefetcher filter is not deleted ;)

I admit that it isn't so obvious as I was thinking in October BUT memory issues are always so ... unpredictable.

[StainlessS]

I'll gladly do, just working on it.



Interesting. I always wondered why identical function_parameter lines appeared more than once. If Groucho2004 extended the AVSInfoTool to display the parameter lists, we'd probably know if it's a bug in AviSynth or in RT_Stats. Actually, I re-included a check that prevents similar lines also for internal functions, not only for plugins (updated my post in RT_Stats thread) though I understand it would be better to list the functions with their multiple parameter lists.

I believe this has come up before, I think that only a single parameter list is available outside of Avisynth dll (available internal only) and so AVSInfoTool will likewise be oblivious to the differing arg list. I take it that the same is true of AVS+, (although I seem to remember that some lists in AVS+ have been removed, ie return empty strings).

[ultim]

ultim

The only question left is how I was able to remove memory leaks with those two patches to AVS+? :)

Just try to believe that cache is not cleaned and prefetcher filter is not deleted ;)

I admit that it isn't so obvious as I was thinking in October BUT memory issues are always so ... unpredictable.

I believe you that cache and prefetcher are not cleaned. I also verified it myself. I'm just saying those not being cleaned is a result of x64 ffdshow doing something different than "normal". At least most probably as things look right now. Like not releasing the clip before the ScriptEnvironment is destroyed, for example, would lead to the exact same leak. Why your patches "solve" this is also pretty clear. They basically inhibit caching, so even if caches are not cleaned up, there is little left to leak. But as you might guess, your patches are not the solution to this problem.

Can someone from you guys please have a look at ffdshow? The main points to watch out for are that
- all clip and frame references are released before the ScriptEnv is destructed,
- that the ScriptEnv is destructed every time the avisynth script is disabled or reloaded, and
- that the latest avs+ header is used for compilation.

The second point is also important to ensure correct script execution, due to plugin loading.

EDIT: Notes on using the correct avisynth header. Avs+'s header is compatible with all previously compiled x64 plugins, even those not compiled for Avs+. So using Avs+'s header makes sure you use the correct one as it is also the latest. The very recent Avisynth 2.6 RC1 introduced modifications to the x64 interface which is NOT compatible to any x64 plugin from before, so until the situation is cleared up, I strongly advice you not to use the Avisynth 2.6 header in 64-bit mode, as it will break any and all 64-bit plugins.

[chainik_svp]

One "patch" was pretty simple. I have a very dumb way to debug something inside AVS and/or plugins - by placing fprintf.
And I was placed them into both Prefetcher constructor and destructor. And guess what? :)
Well, may be I have to re-check it with the latest sources...

About the cache issue.

The only place where VideoFrame can be born is ScriptEnvironment::GetNewFrame().
And it can't be made with some another vfb (unlike how it's working in Avisynth).
In fact every VideoFrame is always have it's own vfb. So it (VideoFrame) should also destroy it's vfb.

Again: vfb's are NOT shared between video frames.

So your main point about
You cannot delete the vfb every time a VideoFrame is destroyed, because due to subframes, vfbs can be shared among multiple VideoFrames.
doesn't seems to be correct.
Or I'm still missing something :)

[chainik_svp]

Can someone from you guys please have a look at ffdshow?

and you DO understand that we can wait for it forever? :)

[ultim]

One "patch" was pretty simple. I have a very dumb way to debug something inside AVS and/or plugins - by placing fprintf.
It's not dumb. A perfectly valid and commonly used method. I use it too, in addition to the CRT debug heap that let's me catch even the smallest leaks that are not visible in the task manager, as well as determine their location.



Again: vfb's are NOT shared between video frames.

So your main point about

doesn't seems to be correct.
Or I'm still missing something :)
Yep, you are. Have a look at VideoFrame::Subframe or ScriptEnv::Subframe.

[chainik_svp]

... and regardless of whether VFBs shared or not, they still have to be unconditionally deleted in ~ScriptEnvironment()

[ultim]

... and regardless of whether VFBs shared or not, they still have to be unconditionally deleted in ~ScriptEnvironment()
They are unconditionally deleted, as soon as the last videoframe pointing to them gets deleted. If I didn't have this single condition, the destructor would crash or cause memory corruption due to double-freeing the same memory area when the vfb is referenced multiple times.

[chainik_svp]

Have a look at VideoFrame::Subframe or ScriptEnv::Subframe.

Good point.

Can you point where exactly in that simplest "BilinearResize" script that SubFrame feature is used? :)
I see now that deleting VFB from VideoFrame will break "sub-frames", BUT was I actually using them?


They are unconditionally deleted

And we're back again.
They are NOT.

[ultim]

Can you point where exactly in that simplest "BilinearResize" script that SubFrame feature is used? :)
But that destructor has to work for all filters! Not just for BilinearResize that happens to not use subframes.



They are unconditionally deleted
And we're back again.
They are NOT.
Please read my previous post to the end. If I deleted vfbs unconditionally, that would crash (or worse) for filters that use subframes. Again, this cleanup has to work for all filters. Not just for filters that You happen to use.

[chainik_svp]

In the simple script that using only internal "BilinearResize" filter I'm definitely NOT using "SubFrame" function. Right?
So there should be no frames created through VideoFrame::Subframe().
And still VideoFrame objects are not cleared because of refcount != 0.

I have no idea, may be it's (SubFrame) is called somewhere internally o_O

[ultim]

In the simple script that using only internal "BilinearResize" filter I'm definitely NOT using "SubFrame" function. Right?
So there should be no frames created through VideoFrame::Subframe().
And still VideoFrame objects are not cleared because of refcount != 0.
Exactly. And this most likely means the host application didn't release the filter chain before destroying the ScriptEnv. Which is why I would like someone to check on ffdshow.
Edit: The reason I find this the most likely reason instead of for example a reference-counting bug in avs+, is because this only happens with a specific version of ffdshow. With any other application, no matter if 32- or 64-bits, this leak could not be observed.

[chainik_svp]

I've got your point in the end :D

It's not very hard to test ffdshow with Avisynth (2.5.8) x64, right? It should give the very same behavior.

And again, we can wait forever for someone to look into ffdshow code.
As I'm the only one who needs it here, I'll do it... :(

[chainik_svp]

I can't find any WIN64 specific code in ffdshow.


struct Tinput : Tavisynth_c {
...
IScriptEnvironment* env;
PClip* clip;
...
~Tinput() {
if (clip) {
delete clip;
}
if (env) {
delete env;
}
}
};


But I'm scared of:

struct Tavisynth_c : public Tdll, w/o destructor at all
...
struct Tinput : Tavisynth_c
...
~Tinput()

I'm done for today...

[innocenat]

Just a quick chime in: ffdshow use Avisynth C interface, with Avisynth API version 2 (according to tryout-trunk). The IScriptEnvironment and PClip are just facade with same interface as the Avs C++ interface, but internally call Avs C interface.

EDIT: There is one mismatch between C interface and C++ interface, mainly in VideoFrame and AVS_VideoFrame structure. It is long vs int problem, though it would only matter if ffdshow is compiled with LP64 data model (not commonly used on windows, but widely on *nix)

[jpsdr]

EDIT: Notes on using the correct avisynth header. Avs+'s header is compatible with all previously compiled x64 plugins, even those not compiled for Avs+. So using Avs+'s header makes sure you use the correct one as it is also the latest. The very recent Avisynth 2.6 RC1 introduced modifications to the x64 interface which is NOT compatible to any x64 plugin from before, so until the situation is cleared up, I strongly advice you not to use the Avisynth 2.6 header in 64-bit mode, as it will break any and all 64-bit plugins.

I didn't take a look at this new interface, but according your saying, it seems that they finaly want to do what they should have done since the begining, and what was done since the begining in VirtualDub : size_t for pointer, and ptrdiff_t for memory offset. But with all the x64 pluggin allready done, this may be a little to late...

Edit : I took a look... It's not even made properly (at least, from my point of view). Some pitch became size_t, when the right choice would be ptrdiff_t, alowing this way positive and negative offset.

[ultim]

Edit : I took a look... It's not even made properly (at least, from my point of view). Some pitch became size_t, when the right choice would be ptrdiff_t, alowing this way positive and negative offset.

That'd be still fine in this case. In Avisynth buffer offsets shouldn't be negative anyway, so here a size_t actually fits better than a ptrdiff_t, because it mirrors the developer's intentions more closely. What bothers me is that they did this for... what reason exactly? This change in the interface has no present or future benefit for Avisynth, yet it breaks all existing 64-bit plugins.

[Stereodude]

This change in the interface has no present or future benefit for Avisynth, yet it breaks all existing 64-bit plugins.
Maybe that was the reason. :scared:

[Wilbert]

Yes sure that was the reason. If it isn't done correct in the past, that's shouldn't be a reason to do that too in the future. I understand it is very inconvienient for the users of existing 64-plugins and i guess for Avisynth+ users too, and apologies for that, but it shouldn't be too much hassle to recompile the important 64-bit plugins. Or do i miss something and is it a lot of work?

[LoRd_MuldeR]

Yes sure that was the reason. If it isn't done correct in the past, that's shouldn't be a reason to do that too in the future. I understand it is very inconvienient for the users of existing 64-plugins and i guess for Avisynth+ users too, and apologies for that, but it shouldn't be too much hassle to recompile the important 64-bit plugins. Or do i miss something and is it a lot of work?

Is there any mechanism that prevents the "old"-style x64 plug-in's from loading with the "new"-style x64 Avisynth, or will they simply cause undefined behavior (crash)?

[ultim]

I'd agree if these changes were any more correct. But you weren't storing pointers in smaller-size integers or doing stupid stuff like that previously, you were just merely limiting the size of an image plane to 2 gigabytes. Since a frame size that large is not in any way realistic, please elaborate how the previous types were more wrong than the new interface. Your reasoning cannot be "because these are pointer offsets", because with that puristical and theoretical argument, the choice of a size_t is equally wrong, as it should be ptrdiff_t then. Fortunately, both size_t and int work perfectly without disadvantages in any real use case. The only practical difference is breaking old plugins or not.

[qyot27]

What would be necessary for (as mentioned in the RC1 thread concerning what VapourSynth does) storing planes in separate pointers, or for >8bit processing? Or both of those together. Because it sounded like it could be needed by the case of using separate pointers per-plane, and if that's a feature that is planned for 2.6 or some future version, that would be a rationale even without high bit depth support.

I'm mostly just curious about what those things would actually need. libavformat handles the frame size calculation issue by casting to int64_t when reading in the width and height, as you can see here (http://git.videolan.org/?p=ffmpeg.git;a=blob;f=libavformat/avisynth.c#l453). If the problem shows up around 9k x 9k @ 8bit, then >8bit would spark it at resolutions that are within cases we might very well see in the near future (like 8K UHD). If it's the same for plugins, anyway (or not a libavformat-specific thing); libavformat is simply taking input from the AviSynth library, not from the plugins.

EDIT: Not that I've tried doing the math to figure out what the maximum allowable frame sizes in a 2GB plane case is. I'm mostly just thinking out loud.

[ultim]

Storing planes in separate pointers like VS is purely a design decision, and has zero effect on the supported feature set. At most it allows some negligible performance optimizations. The most important thing that Avisynth would need to support more features is to be able to store additional data members in Clip and VideoFrame classes. In a clean solution, that would also be a breaking change to plugins. But even this is solvable without breaking plugins, using hacks in the code and additional complexity, but definitely possible. For supporting >8bit, not even that is necessary, the main thing inhibiting higher bitdepths is upgrading all the plugins and internal functions to work with >8bits: a crazy amount of coding necessary, lot of time, and relatively few people capable at all.

As for handling UHD: a full 9k x 9k frame, even if you take something very odd, like 16bits, no subsampling, RGB interleaved, with alpha transparency (this is about as large as you can get), still only consumes about 618 Mbytes of memory. So even for something crazy like this, the old datatypes were more than enough. So you cannot justify breaking the interface by that.

[ultim]

Is there any mechanism that prevents the "old"-style x64 plug-in's from loading with the "new"-style x64 Avisynth, or will they simply cause undefined behavior (crash)?

Undefined behavior.

[TheFluff]

I think you are a bit overly attached to an ABI from like 2002. Especially since almost nobody actually uses 64-bit Avisynth (there was a poll about this a year back or so, IIRC basically nobody claimed to be using 64-bit Avisynth). I was about to say this is a golden opportunity to finally man up and fix that hilariously retarded plugin interface, but then I realized this is Avisynth we're talking about and obviously everything needs to work just like in the 90's or people will hurf a durf.

[ultim]

I think you are a bit overly attached to an ABI from like 2002. Especially since almost nobody actually uses 64-bit Avisynth (there was a poll about this a year back or so, IIRC basically nobody claimed to be using 64-bit Avisynth). I was about to say this is a golden opportunity to finally man up and fix that hilariously retarded plugin interface, but then I realized this is Avisynth we're talking about and obviously everything needs to work just like in the 90's or people will hurf a durf.

Yes, if you are going to break the interface anyway, just man up as you say and do it correctly, and correct all the other mistakes too. Oh wait, you cannot, because then for those other changes you'd have to break to 32-bit interface too, and that interface /is/ used by a lot of people.

You make it sound like I just want to stay by the old interface no matter what. Wrong! I am not against this change because it touches some old stuff. I am against it because it doesn't bring any kind of benefit, but causes problems. If there is a good reason, like correcting design mistakes, modernizing it, adding new features, or anything along these lines, I'd just go with it. But this change was none of them, yet makes all plugins from the past 6+ years unusable.

Avisynth+ will break the interface in the future, moreoever, it will do it drastically. It is already being planned. But, very importantly, it will do it only once, for real benefits and progress, and will do it in a sustainable way such that breaking modification won't be necessary again after that. And I am waiting for it eagerly, and I won't shed a single tear for the old interface. But this needs preparation, both on Avisynth's core's side as well on the plugins' side.

Make it personal if you really want to. But in the end, every real programmer should see this is not about "manning up", or throwing false accusations around that I just love stuff from the 90's. It is about careful preparation, waiting out until stuff is ready, then doing the change responsibly in the end.

[Stereodude]

Especially since almost nobody actually uses 64-bit Avisynth (there was a poll about this a year back or so, IIRC basically nobody claimed to be using 64-bit Avisynth).
I use it extensively along with MP_Pipeline for mixing 32-bit and 64-bit. Even if all the scripting is really in 32-bit at the very least its very handy to feed x264 64-bit without having to pipe it.

[jpsdr]

Especially since almost nobody actually uses 64-bit Avisynth
ME !!! ME !!! ME !!!
I am, i am, i am !!!
;)

And i love stuff from 90's, even 80's, it was my youth.:p

[kaefert]

I also use 64bit, since with avisynth 32bit my scripts simply won't work at all. I'm using avisynth as a non-linear video editor therefore my scripts have A LOT of input streams which simply bursts the memory limitations of 32bit.

[ultim]

So you are writing your own AviSynth plugin (cool!), and obviously one of the first things you have to do in your code is to include the AviSynth header. But which one? With all the different header variants lying around it is easy to get lost if you haven't been following AviSynth's development for a long time. Should you copy the header from another plugin? Should you copy it from the AviSynth64 project to be 64-bits compatible? Should you take the 2.5 header as it is the latest release that is officially stable? Do you need SEt's AviSynth-MT header if you want multithreading compatibility? Do you need seperate headers for 32- and 64-bits like most plugins ship it? Should you just take the latest header from the AviSynth 2.6 project? And what about AviSynth+'s header?

Fortunately, no matter how you answer the above questions, there is one (and just one) solution that is easy to implement and fits all needs: Use AviSynth+'s header. And if you'd like to know why, read on.

So let's tackle the above questions.

Should you copy the header from another plugin?
No. Most plugins are older then AviSynth project releases, and so they ship with outdated (and sometimes buggy) headers. Also, some plugins have both separate 32- and 64-bit sources, so you still wouldn't know which one to take. And if you are really unlucky, you might stumble on a plugin that was written for AviSynth 2.5, and using that header would be the worst of all your header-related options.

Should you take the 2.5 header as it is the latest release that is officially stable?
No. 2.5 is no more. Most plugins that have originally been written for 2.5 have been already recompiled for 2.6. Don't try to be smart and support both versions, because they are not compatible. 2.6 has been around for many years now, and the existing plugin ecosystem builds exstensively around this version. Technically speaking, it is stable. Nobody uses 2.5 any more.

Should you copy it from the AviSynth64 project to be 64-bits compatible?
No. While AviSynth64's header will work perfectly if you want your plugin to *only* run in 64-bit mode, that is most likely not the case. That project isn't maintained any more, and thanks to that the 32-bit part is out of date.

Do you need seperate headers for 32- and 64-bits like most plugins ship it?
No. You will see plugins around that have both avisynth.h and avisynth64.h. Same for many applications hosting avisynth.dll. This is because the original AviSynth project never supported 64-bit processing (not even today), so these other projects took the 32-bit header from the latest AviSynth version that was available when they were created, and they took the 64-bit header from the AviSynth64 project. This resulted in an ecosystem where the 64-bit versions didn't see any improvements over the years. On the upside, avisynth64.h stayed stable. On the downside, the 32-bit and 64-bit headers started drifting apart. Nevertheless, a merge of the avisynth.h and avisynth64.h headers is easily possible, which is exactly what AviSynth+ has done. There is no need for two separate headers, it only results in additional code, complexity, and maintenance burden.

Do you need AviSynth-MT's header if you want multithreading compatibility?
No. While properly supporting multithreaded versions does require special coding considerations from plugin writers (see this other article (http://forum.doom9.org/showthread.php?p=1649886#post1649886)), none of those considerations affect the choice of header. There is no API or ABI difference between multi-threaded and single-threaded AviSynth versions. You can perfectly support MT-capable AviSynth versions even if using the header from an AviSynth variant that has no MT-support.

Should you just take the latest header from the AviSynth 2.6 project?
No. This project (sometimes people refer to it as the "original" or "official" AviSynth, though somewhat incorrect (http://forum.doom9.org/showthread.php?p=1693944#post1693944)) always has the latest version, but it will do you no good if you want to support 64-bit processing. You cannot compile your plugin using its header in 64-bit mode, which is why people started using avisynth64.h in the first place. Even if it decided to support 64-bit in the future, it wouldn't be compatible to the existing (and pretty large) 64-bit ecosystem anymore, throwing away all the 64-bit plugin and application development that has been done in the past 6 years or so. And as already said, using two separate headers is completely unnecessary and only leads to additional complications down the road.

What about AviSynth+'s header?
The headers of AviSynth+ (https://github.com/AviSynth/AviSynthPlus/tree/MT/avs_core/include) are up to date in every aspect and provide the greatest possible compatibility. By using AviSynth+'s headers, applications and plugins can cleanly compile and run in 32-bits and 64-bits. It is 100% compatible to the latest 32-bit development on the old AviSynth 2.6 project, while supporting all 64-bit binaries. And of course, you can use it regardless if you support multithreading or not. Furthermore and importantly, it is fully compatible to installations of the AviSynth 2.6, AviSynth-MT, AviSynth64, and of course the AviSynth+ projects, so your plugin/application will be able to run on any user's machine.

[chainik_svp]

ultim
Use AviSynth+'s header.

So why it still contains that "normal licence conditions will be reapplied in a future version" thing from AVS 2.6 header? :)

[ultim]

ultim
Use AviSynth+'s header.

So why it still contains that "normal licence conditions will be reapplied in a future version" thing from AVS 2.6 header? :)

The 2.6 header is originally written by Ianb, and even though we have made a derivative work of it, we cannot legally relicense the header without his explicit permission. LGPL-like condiitons (even if technically not LGPL) will be restored when 2.6 exits the RC stage, or when Avisynth+ creates a 100% new header, whichever comes first.

[chainik_svp]

So actually there's still a room for good old 2.5 headers...

// Avisynth v2.6. Copyright 2006 Klaus Post.
// Avisynth v2.6. Copyright 2009 Ian Brabham.
What will come first - GRRM will finish "Song of ice and fire" or Avisynth 2.6 will exit RC stage?

[Reel.Deel]

Especially since almost nobody actually uses 64-bit Avisynth (there was a poll about this a year back or so, IIRC basically nobody claimed to be using 64-bit Avisynth).

Maybe because that thread (http://forum.doom9.org/showthread.php?t=168478) was created before AviSynth+ came into fruition. Also 64-bit avs was not even an option. I'm sure the number of users has grown since AviSynth+.

[ultim]

So actually there's still a room for good old 2.5 headers...


What will come first - GRRM will finish "Song of ice and fire" or Avisynth 2.6 will exit RC stage?

Of course, this is only a problem if you want to distribute a closed-source application or filter. Closed source binaries must use a header from before Aug 14, 2009.

[ultim]

Maybe because that thread (http://forum.doom9.org/showthread.php?t=168478) was created before AviSynth+ came into fruition. Also 64-bit avs was not even an option. I'm sure the number of users has grown since AviSynth+.

Yeah, it's kind of hard to vote for something if it isn't even an option in the poll. LOL

[martin53]

Oh my gosh, I bet it has to do something with AviSynth's StringDump. This is only a hunch though. Is it reproducible on classic Avisynth, even if only sporadically?

I instrumented the script to report the processing of most lines via DbgView, see embedded code. This script ran several dozen times with 2.6 MT without fail in patched AvsPmod. After I lost patience today and installed Avisynth+ r1576, AvsPmod crashed in several runs and paused (maybe better term than freeze here) in all other runs, between line 124 and 125. Try {RT_Version()} catch (msg) {assert(false,"This script requires the RT_Stats plugin.")}
Try {GEval("")} catch (msg) {assert(false,"This script requires the GScript plugin.")}
global ofs=0

#============================================================================
Filename=RT_FSelSaveAs(title="Save Filter list",dir="",filt="Text files|*.txt",fn="_All+Functions_.txt",debug=true)
Assert(FileName.IsString(), "Please provide a file name to save the filter list")

#============================================================================
RT_WriteFile(Filename, VersionString(), Append=false) Try {assert(false)} catch(err) {Line(err,0)}
WriteInternalFunctionsSyntax(RT_InternalFunctions(), Filename) Try {assert(false)} catch(err) {Line(err,0)}
WritePluginFunctionsSyntaxBackward(RT_PluginFunctions(), Filename) Try {assert(false)} catch(err) {Line(err,0)}
WritePluginFunctionsSyntax(RT_PluginFunctions(), Filename) Try {assert(false)} catch(err) {Line(err,0)}
WritePluginFunctionsSyntaxBackward(RT_PluginFunctions(), Filename) Try {assert(false)} catch(err) {Line(err,0)}
WritePluginFunctionsSyntax(RT_PluginFunctions(), Filename) Try {assert(false)} catch(err) {Line(err,0)}
WritePluginFunctionsSyntaxBackward(RT_PluginFunctions(), Filename) Try {assert(false)} catch(err) {Line(err,0)}
WritePluginFunctionsSyntax(RT_PluginFunctions(), Filename) Try {assert(false)} catch(err) {Line(err,0)}
WritePluginFunctionsSyntaxBackward(RT_PluginFunctions(), Filename) Try {assert(false)} catch(err) {Line(err,0)}
BlankClip().subtitle("Function info is complete.",align=5)

#============================================================================
function WriteInternalFunctionsSyntax(string fu_list, string Filename) {
Try {assert(false)} catch(err) {Line(err,0)}
GScript("""
done="" Try {assert(false)} catch(err) {Line(err,23)}
while(fu_list != "") {
i = FindStr(fu_list+" ", " ") Try {assert(false)} catch(err) {Line(err,23)}
fu_name = LeftStr(fu_list, i-1) Try {assert(false)} catch(err) {Line(err,23)}
fu_list = MidStr(fu_list, i+1) Try {assert(false)} catch(err) {Line(err,23)}
params = RT_PluginParam(fu_name) Try {assert(false)} catch(err) {Line(err,23)}
if (FindStr(done, fu_name+params)==0) {
done = done + " "+fu_name+params Try {assert(false)} catch(err) {Line(err,23)}
RT_WriteFile(Filename, "internal %s ( %s ) -> %s", fu_name, params, GetReturnType(fu_name, params), Append=true)
Try {assert(false)} catch(err) {Line(err,23)}
}
}
""")
}
#============================================================================
function WritePluginFunctionsSyntax(string fu_list, string Filename) {
Try {assert(false)} catch(err) {Line(err,0)}
GScript("""
done="" Try {assert(false)} catch(err) {Line(err,41)}
while(fu_list != "") {
i = FindStr(fu_list+" ", " ") Try {assert(false)} catch(err) {Line(err,41)}
fu_name = LeftStr(fu_list, i-1) Try {assert(false)} catch(err) {Line(err,41)}
fu_list = MidStr(fu_list, i+1) Try {assert(false)} catch(err) {Line(err,41)}
i = FindStr(fu_name, "_") Try {assert(false)} catch(err) {Line(err,41)}
if (i>0) {
params = RT_PluginParam(fu_name) Try {assert(false)} catch(err) {Line(err,41)}
if (FindStr(fu_list, "_"+fu_name)==0 && FindStr(done, "_"+fu_name+params)==0) {
done = done + " _"+fu_name+params Try {assert(false)} catch(err) {Line(err,41)}
RT_WriteFile(Filename, "plugin %s ( %s ) -> %s", fu_name, params, GetReturnType(fu_name, params), Append=true)
Try {assert(false)} catch(err) {Line(err,41)}
}
}
}
""")
}
#============================================================================
function WritePluginFunctionsSyntaxBackward(string fu_list, string Filename) {
Try {assert(false)} catch(err) {Line(err,0)}
GScript("""
done="" Try {assert(false)} catch(err) {Line(err,62)}
fu_list = RevStr(fu_list) Try {assert(false)} catch(err) {Line(err,62)}
while(fu_list != "") {
i = FindStr(fu_list+" ", " ") Try {assert(false)} catch(err) {Line(err,62)}
fu_name = RevStr(LeftStr(fu_list, i-1)) Try {assert(false)} catch(err) {Line(err,62)}
fu_list = MidStr(fu_list, i+1) Try {assert(false)} catch(err) {Line(err,62)}
i = FindStr(fu_name, "_") Try {assert(false)} catch(err) {Line(err,62)}
if (i>0) {
params = RT_PluginParam(fu_name) Try {assert(false)} catch(err) {Line(err,62)}
if (FindStr(fu_list, "_"+fu_name)==0 && FindStr(done, "_"+fu_name+params)==0) {
done = done + " _"+fu_name+params Try {assert(false)} catch(err) {Line(err,62)}
RT_WriteFile(Filename, "plugin %s ( %s ) -> %s", fu_name, params, GetReturnType(fu_name, params), Append=true)
Try {assert(false)} catch(err) {Line(err,62)}
}
}
}
""")
}
#============================================================================
#function GetReturnType(string fu, string params) { r = "?" }
#============================================================================
function GetReturnType(string fu, string params) {
Try {assert(false)} catch(err) {Line(err,0)}
GScript("""
RT_DebugF("Fu=%s %s", fu, params, name="GetReturnType") Try {assert(false)} catch(err) {Line(err,86)}
if (MidStr(fu,3,1)=="+") {fu = LeftStr(fu,2)+MidStr(fu,2) Try {assert(false)} catch(err) {Line(err,86)}}
if (MidStr(fu,2,1)=="+") {fu = LeftStr(fu,1)+fu Try {assert(false)} catch(err) {Line(err,86)}}
else if (fu=="Assert") {r = "conditional halt" Try {assert(false)} catch(err) {Line(err,86)}}
else if (FindStr(fu, "avss_26_DSS2")>0) {r = "clip" Try {assert(false)} catch(err) {Line(err,86)}}
else if (FindStr(fu, "ImageSeq_")>0) {r = "clip" Try {assert(false)} catch(err) {Line(err,86)}}
else if (FindStr(fu, "LoadCPlugin")>0) {r = "void" Try {assert(false)} catch(err) {Line(err,86)}}
else if (FindStr(fu, "LoadVirtualdubPlugin")>0) {r = "void" Try {assert(false)} catch(err) {Line(err,86)}}
else if (FindStr(fu, "MDepan")>0) {r = "clip" Try {assert(false)} catch(err) {Line(err,86)}}
else if (FindStr(fu, "MSCDetection")>0) {r = "clip" Try {assert(false)} catch(err) {Line(err,86)}}
else if (FindStr(fu, "mvtools2_")>0) {r = "clip?" Try {assert(false)} catch(err) {Line(err,86)}}
else if (FindStr(fu, "ResampleAudio")>0) {r = "clip" Try {assert(false)} catch(err) {Line(err,86)}}
else if (FindStr(fu, "Resize")>0) {r = "clip" Try {assert(false)} catch(err) {Line(err,86)}}
else if (FindStr(fu, "RT_Debug")>0) {r = "clip" Try {assert(false)} catch(err) {Line(err,86)}}
else if (FindStr(fu, "RT_FSel")>0) {r = "." Try {assert(false)} catch(err) {Line(err,86)}}
else if (FindStr(fu, "Source")>0) {r = "clip" Try {assert(false)} catch(err) {Line(err,86)}}
else {
bc = BlankClip(width=8,height=8,length=1,pixel_type="YV12")
Try {assert(false)} catch(err) {Line(err,86)}
fp = LeftStr(params,2)=="cc" ? "(bc,bc)"
\ : LeftStr(params,2)=="cf" ? "(bc,0.1)"
\ : LeftStr(params,3)=="cci" ? "(bc,bc,0)"
\ : LeftStr(params,5)=="ciiii" ? "(bc,0,0,0,0)"
\ : LeftStr(params,4)=="ciii" ? "(bc,0,0,0)"
\ : LeftStr(params,3)=="cii" ? "(bc,0,0)"
\ : LeftStr(params,2)=="ci" ? "(bc,0)"
\ : LeftStr(params,1)=="c" ? "(bc)"
\ : LeftStr(params,2)=="ff" ? "(0.1,0.1)"
\ : LeftStr(params,1)=="f" ? "(0.1)"
\ : LeftStr(params,2)=="ii" ? "(0,0)"
\ : LeftStr(params,1)=="i" ? "(0)"
\ : LeftStr(params,2)=="ss" ? "("+chr(34)+"1"+chr(34)+","+chr(34)+"1"+chr(34)+")"
\ : LeftStr(params,2)=="si" ? "("+chr(34)+"1"+chr(34)+",1)"
\ : LeftStr(params,1)=="s" ? "("+chr(34)+"1"+chr(34)+")"
\ : "()" Try {assert(false)} catch(err) {Line(err,86)}
Try {
Try {assert(false)} catch(err) {Line(err,86)}
r = Eval("current_frame=0 "+fu+fp) Try {assert(false)} catch(err) {Line(err,86)}
r = r.IsBool ? "bool" : r.IsClip ? "clip" : r.IsInt ? "int" : r.IsFloat ? "float" : r.IsString ? "string" : "?"
Try {assert(false)} catch(err) {Line(err,86)}
} catch (msg) {
Try {
r = r.IsBool ? "bool" : r.IsClip ? "clip" : r.IsInt ? "int" : r.IsFloat ? "float" : r.IsString ? "string" : "?"
} catch(msg) {
r = "?" Try {assert(false)} catch(err) {Line(err,86)}
}
}
if (r=="?") {
bc = BlankClip(width=8,height=8,length=1,pixel_type="RGB32")
Try {assert(false)} catch(err) {Line(err,86)}
fp = LeftStr(params,2)=="cc" ? "(bc,bc)"
\ : LeftStr(params,2)=="cf" ? "(bc,0.1)"
\ : LeftStr(params,3)=="cci" ? "(bc,bc,0)"
\ : LeftStr(params,5)=="ciiii" ? "(bc,0,0,-0,-0)"
\ : LeftStr(params,4)=="ciii" ? "(bc,0,0,0)"
\ : LeftStr(params,3)=="cii" ? "(bc,0,0)"
\ : LeftStr(params,2)=="ci" ? "(bc,0)"
\ : "(bc)" Try {assert(false)} catch(err) {Line(err,86)}
Try {
Try {assert(false)} catch(err) {Line(err,86)}
r = Eval("current_frame=0 "+fu+fp) Try {assert(false)} catch(err) {Line(err,86)}
r = r.IsBool ? "bool" : r.IsClip ? "clip" : r.IsInt ? "int " : r.IsFloat ? "float" : r.IsString ? "string" : "?"
Try {assert(false)} catch(err) {Line(err,86)}
} catch (msg) {
r = "?" Try {assert(false)} catch(err) {Line(err,86)}
}
}
if (r=="?" && LeftStr(params,1)=="c") {r="clip?" Try {assert(false)} catch(err) {Line(err,86)}}
}
""")
Try {assert(false)} catch(err) {Line(err,0)}
return r
}
#=====================================================================================================================
function Line(string s, int ofs) {ofs=default(ofs,0) s=RightStr(s,FindStr(RevStr(s)," ")-1) s=LeftStr(s,StrLen(s)-1) l=int(value(s)+ofs) RT_DebugF("%d",l,name="@line ")}
#=====================================================================================================================

Script may look complicated first but is easy to understand I think. Best I can do today is provide this 'tool' to provoke the failure.
I do assume that the behavior is machine-dependent, since FFT plugins make it more prone to fail,
When reproducing, note that line numbers are logged after the command in that line.

[qyot27]

With the recent changes to how the version information is generated internally, was it intentional that the output of Version() itself doesn't use AVS_FULLVERSION now, but only AVS_PROJECT and AVS_COPYRIGHT?

[ultim]

With the recent changes to how the version information is generated internally, was it intentional that the output of Version() itself doesn't use AVS_FULLVERSION now, but only AVS_PROJECT and AVS_COPYRIGHT?

Of course not. Fixed :D

[jpsdr]

@Ultim
I've updated as you adviced my avisynth.h to your's last version.
I've been obliged to make the following changes :
Replace all the

<avs/xxx.h>

with

"./avs/xxx.h"

For a redistribuable header for creating pluggins, i think it would be better to use "" version instead of <>.

My real question indeed, is : When i tried to recompile my filters, i have one which use "CACHE_25_RANGE". I guess remplacement should be "CACHE_GET_RANGE", but i want to be sure. Thanks.

The exact code is :

child->SetCacheHints(CACHE_25_RANGE,3);
.....
v.AsClip()->SetCacheHints(CACHE_25_RANGE,2);

[innocenat]

I think it was decided (for Avs+), that you should include $AVISYNTH_INSTALL_DIR/include to your project include directory inatead of copy them everywhere.

[jpsdr]

That's work only if you have avs+ installed, maybe not if you have avs2.6.
Well... it realy doesn't matter, this issue is not a big deal.

[ultim]

My real question indeed, is : When i tried to recompile my filters, i have one which use "CACHE_25_RANGE". I guess remplacement should be "CACHE_GET_RANGE", but i want to be sure. Thanks.

The exact code is :

child->SetCacheHints(CACHE_25_RANGE,3);
.....
v.AsClip()->SetCacheHints(CACHE_25_RANGE,2);


No, that is actually the wrong fix. The old CACHE_25_* constants got removed by IanB. No plugin should use them anymore. For replacement, you can use one of the first four members of the CachePolicyHint enum. He has documented his new semantics in this post (http://forum.doom9.org/showthread.php?p=1595750#post1595750). According to his explanations, the correct replacement in your case should be CACHE_WINDOW, but please also read his post.

[ultim]

I think it was decided (for Avs+), that you should include $AVISYNTH_INSTALL_DIR/include to your project include directory inatead of copy them everywhere.

It is $(AVISYNTH_SDK_PATH)\include that should be in your project. And if you don't have Avisynth+'s FilterSDK installed, you can add the path that you manually copied the headers to.

While discussing this on the IRC just recently, it was noticed the installer isn't setting AVISYNTH_SDK_PATH at all. Something that slipped through the cracks. This will be fixed in the next installer on our side.

[jpsdr]

I think i'll keep what i've done. At least, i'm sure if someone get my project from github :
- He can compile it whatever avisynth version is installed.
- I'm sure it will compile with the same files as me, thing not sure with the way you describe.

Anyway, thanks for your return and these informations.

[innocenat]

It was also discussed that if each plugin keeps its own avisynth.h, when people are compiling it they need to know what version of avisynth.h it is: 2.5? 2.6-a3? 2.6a5? 2.6rc1? 2.6-64bit? Avs+? Even though Avs+ header is compatible with 2.6a5, 2.6rc1 and 2.6MT, in future if breaking change happen (like 2.6rc1 64bit interface), people who are recompiling your plugin need to manually look what version of avisynth.h you are using.

Though personally I also find bundling avisynth.h much easier

[jpsdr]

There is, from my point of view, no perfect solution. If people have 2.6rc1, if they build 64 bit version, compatibility is broken. If they have 2.5 installed or a too old 2.6, it will (or may) not compile at all...
After, personnal taste, i'm not big fan of "Getting my project is not enough, you have also to install X,Y,Z,W, etc... if you want to build it" (in fact, i realy don't like these kind of situations, but unfortunately sometimes it's realy unavoiable), this is why, i personnaly choose the "No need to install anything" solution package.
So, for me, both solutions have pro and cons. I choose what i personnaly prefer, because, i like less the other option.
After, i'm absolutely not trying to argue to change anything. Things have been explained, for valid reasons, it's totaly Ok for me.

[ultim]

I instrumented the script to report the processing of most lines via DbgView, see embedded code. This script ran several dozen times with 2.6 MT without fail in patched AvsPmod. After I lost patience today and installed Avisynth+ r1576, AvsPmod crashed in several runs and paused (maybe better term than freeze here) in all other runs, between line 124 and 125. Try {RT_Version()} catch (msg) {assert(false,"This script requires the RT_Stats plugin.")}
Try {GEval("")} catch (msg) {assert(false,"This script requires the GScript plugin.")}
global ofs=0

#============================================================================
Filename=RT_FSelSaveAs(title="Save Filter list",dir="",filt="Text files|*.txt",fn="_All+Functions_.txt",debug=true)
Assert(FileName.IsString(), "Please provide a file name to save the filter list")

#============================================================================
RT_WriteFile(Filename, VersionString(), Append=false) Try {assert(false)} catch(err) {Line(err,0)}
WriteInternalFunctionsSyntax(RT_InternalFunctions(), Filename) Try {assert(false)} catch(err) {Line(err,0)}
WritePluginFunctionsSyntaxBackward(RT_PluginFunctions(), Filename) Try {assert(false)} catch(err) {Line(err,0)}
WritePluginFunctionsSyntax(RT_PluginFunctions(), Filename) Try {assert(false)} catch(err) {Line(err,0)}
WritePluginFunctionsSyntaxBackward(RT_PluginFunctions(), Filename) Try {assert(false)} catch(err) {Line(err,0)}
WritePluginFunctionsSyntax(RT_PluginFunctions(), Filename) Try {assert(false)} catch(err) {Line(err,0)}
WritePluginFunctionsSyntaxBackward(RT_PluginFunctions(), Filename) Try {assert(false)} catch(err) {Line(err,0)}
WritePluginFunctionsSyntax(RT_PluginFunctions(), Filename) Try {assert(false)} catch(err) {Line(err,0)}
WritePluginFunctionsSyntaxBackward(RT_PluginFunctions(), Filename) Try {assert(false)} catch(err) {Line(err,0)}
BlankClip().subtitle("Function info is complete.",align=5)

#============================================================================
function WriteInternalFunctionsSyntax(string fu_list, string Filename) {
Try {assert(false)} catch(err) {Line(err,0)}
GScript("""
done="" Try {assert(false)} catch(err) {Line(err,23)}
while(fu_list != "") {
i = FindStr(fu_list+" ", " ") Try {assert(false)} catch(err) {Line(err,23)}
fu_name = LeftStr(fu_list, i-1) Try {assert(false)} catch(err) {Line(err,23)}
fu_list = MidStr(fu_list, i+1) Try {assert(false)} catch(err) {Line(err,23)}
params = RT_PluginParam(fu_name) Try {assert(false)} catch(err) {Line(err,23)}
if (FindStr(done, fu_name+params)==0) {
done = done + " "+fu_name+params Try {assert(false)} catch(err) {Line(err,23)}
RT_WriteFile(Filename, "internal %s ( %s ) -> %s", fu_name, params, GetReturnType(fu_name, params), Append=true)
Try {assert(false)} catch(err) {Line(err,23)}
}
}
""")
}
#============================================================================
function WritePluginFunctionsSyntax(string fu_list, string Filename) {
Try {assert(false)} catch(err) {Line(err,0)}
GScript("""
done="" Try {assert(false)} catch(err) {Line(err,41)}
while(fu_list != "") {
i = FindStr(fu_list+" ", " ") Try {assert(false)} catch(err) {Line(err,41)}
fu_name = LeftStr(fu_list, i-1) Try {assert(false)} catch(err) {Line(err,41)}
fu_list = MidStr(fu_list, i+1) Try {assert(false)} catch(err) {Line(err,41)}
i = FindStr(fu_name, "_") Try {assert(false)} catch(err) {Line(err,41)}
if (i>0) {
params = RT_PluginParam(fu_name) Try {assert(false)} catch(err) {Line(err,41)}
if (FindStr(fu_list, "_"+fu_name)==0 && FindStr(done, "_"+fu_name+params)==0) {
done = done + " _"+fu_name+params Try {assert(false)} catch(err) {Line(err,41)}
RT_WriteFile(Filename, "plugin %s ( %s ) -> %s", fu_name, params, GetReturnType(fu_name, params), Append=true)
Try {assert(false)} catch(err) {Line(err,41)}
}
}
}
""")
}
#============================================================================
function WritePluginFunctionsSyntaxBackward(string fu_list, string Filename) {
Try {assert(false)} catch(err) {Line(err,0)}
GScript("""
done="" Try {assert(false)} catch(err) {Line(err,62)}
fu_list = RevStr(fu_list) Try {assert(false)} catch(err) {Line(err,62)}
while(fu_list != "") {
i = FindStr(fu_list+" ", " ") Try {assert(false)} catch(err) {Line(err,62)}
fu_name = RevStr(LeftStr(fu_list, i-1)) Try {assert(false)} catch(err) {Line(err,62)}
fu_list = MidStr(fu_list, i+1) Try {assert(false)} catch(err) {Line(err,62)}
i = FindStr(fu_name, "_") Try {assert(false)} catch(err) {Line(err,62)}
if (i>0) {
params = RT_PluginParam(fu_name) Try {assert(false)} catch(err) {Line(err,62)}
if (FindStr(fu_list, "_"+fu_name)==0 && FindStr(done, "_"+fu_name+params)==0) {
done = done + " _"+fu_name+params Try {assert(false)} catch(err) {Line(err,62)}
RT_WriteFile(Filename, "plugin %s ( %s ) -> %s", fu_name, params, GetReturnType(fu_name, params), Append=true)
Try {assert(false)} catch(err) {Line(err,62)}
}
}
}
""")
}
#============================================================================
#function GetReturnType(string fu, string params) { r = "?" }
#============================================================================
function GetReturnType(string fu, string params) {
Try {assert(false)} catch(err) {Line(err,0)}
GScript("""
RT_DebugF("Fu=%s %s", fu, params, name="GetReturnType") Try {assert(false)} catch(err) {Line(err,86)}
if (MidStr(fu,3,1)=="+") {fu = LeftStr(fu,2)+MidStr(fu,2) Try {assert(false)} catch(err) {Line(err,86)}}
if (MidStr(fu,2,1)=="+") {fu = LeftStr(fu,1)+fu Try {assert(false)} catch(err) {Line(err,86)}}
else if (fu=="Assert") {r = "conditional halt" Try {assert(false)} catch(err) {Line(err,86)}}
else if (FindStr(fu, "avss_26_DSS2")>0) {r = "clip" Try {assert(false)} catch(err) {Line(err,86)}}
else if (FindStr(fu, "ImageSeq_")>0) {r = "clip" Try {assert(false)} catch(err) {Line(err,86)}}
else if (FindStr(fu, "LoadCPlugin")>0) {r = "void" Try {assert(false)} catch(err) {Line(err,86)}}
else if (FindStr(fu, "LoadVirtualdubPlugin")>0) {r = "void" Try {assert(false)} catch(err) {Line(err,86)}}
else if (FindStr(fu, "MDepan")>0) {r = "clip" Try {assert(false)} catch(err) {Line(err,86)}}
else if (FindStr(fu, "MSCDetection")>0) {r = "clip" Try {assert(false)} catch(err) {Line(err,86)}}
else if (FindStr(fu, "mvtools2_")>0) {r = "clip?" Try {assert(false)} catch(err) {Line(err,86)}}
else if (FindStr(fu, "ResampleAudio")>0) {r = "clip" Try {assert(false)} catch(err) {Line(err,86)}}
else if (FindStr(fu, "Resize")>0) {r = "clip" Try {assert(false)} catch(err) {Line(err,86)}}
else if (FindStr(fu, "RT_Debug")>0) {r = "clip" Try {assert(false)} catch(err) {Line(err,86)}}
else if (FindStr(fu, "RT_FSel")>0) {r = "." Try {assert(false)} catch(err) {Line(err,86)}}
else if (FindStr(fu, "Source")>0) {r = "clip" Try {assert(false)} catch(err) {Line(err,86)}}
else {
bc = BlankClip(width=8,height=8,length=1,pixel_type="YV12")
Try {assert(false)} catch(err) {Line(err,86)}
fp = LeftStr(params,2)=="cc" ? "(bc,bc)"
\ : LeftStr(params,2)=="cf" ? "(bc,0.1)"
\ : LeftStr(params,3)=="cci" ? "(bc,bc,0)"
\ : LeftStr(params,5)=="ciiii" ? "(bc,0,0,0,0)"
\ : LeftStr(params,4)=="ciii" ? "(bc,0,0,0)"
\ : LeftStr(params,3)=="cii" ? "(bc,0,0)"
\ : LeftStr(params,2)=="ci" ? "(bc,0)"
\ : LeftStr(params,1)=="c" ? "(bc)"
\ : LeftStr(params,2)=="ff" ? "(0.1,0.1)"
\ : LeftStr(params,1)=="f" ? "(0.1)"
\ : LeftStr(params,2)=="ii" ? "(0,0)"
\ : LeftStr(params,1)=="i" ? "(0)"
\ : LeftStr(params,2)=="ss" ? "("+chr(34)+"1"+chr(34)+","+chr(34)+"1"+chr(34)+")"
\ : LeftStr(params,2)=="si" ? "("+chr(34)+"1"+chr(34)+",1)"
\ : LeftStr(params,1)=="s" ? "("+chr(34)+"1"+chr(34)+")"
\ : "()" Try {assert(false)} catch(err) {Line(err,86)}
Try {
Try {assert(false)} catch(err) {Line(err,86)}
r = Eval("current_frame=0 "+fu+fp) Try {assert(false)} catch(err) {Line(err,86)}
r = r.IsBool ? "bool" : r.IsClip ? "clip" : r.IsInt ? "int" : r.IsFloat ? "float" : r.IsString ? "string" : "?"
Try {assert(false)} catch(err) {Line(err,86)}
} catch (msg) {
Try {
r = r.IsBool ? "bool" : r.IsClip ? "clip" : r.IsInt ? "int" : r.IsFloat ? "float" : r.IsString ? "string" : "?"
} catch(msg) {
r = "?" Try {assert(false)} catch(err) {Line(err,86)}
}
}
if (r=="?") {
bc = BlankClip(width=8,height=8,length=1,pixel_type="RGB32")
Try {assert(false)} catch(err) {Line(err,86)}
fp = LeftStr(params,2)=="cc" ? "(bc,bc)"
\ : LeftStr(params,2)=="cf" ? "(bc,0.1)"
\ : LeftStr(params,3)=="cci" ? "(bc,bc,0)"
\ : LeftStr(params,5)=="ciiii" ? "(bc,0,0,-0,-0)"
\ : LeftStr(params,4)=="ciii" ? "(bc,0,0,0)"
\ : LeftStr(params,3)=="cii" ? "(bc,0,0)"
\ : LeftStr(params,2)=="ci" ? "(bc,0)"
\ : "(bc)" Try {assert(false)} catch(err) {Line(err,86)}
Try {
Try {assert(false)} catch(err) {Line(err,86)}
r = Eval("current_frame=0 "+fu+fp) Try {assert(false)} catch(err) {Line(err,86)}
r = r.IsBool ? "bool" : r.IsClip ? "clip" : r.IsInt ? "int " : r.IsFloat ? "float" : r.IsString ? "string" : "?"
Try {assert(false)} catch(err) {Line(err,86)}
} catch (msg) {
r = "?" Try {assert(false)} catch(err) {Line(err,86)}
}
}
if (r=="?" && LeftStr(params,1)=="c") {r="clip?" Try {assert(false)} catch(err) {Line(err,86)}}
}
""")
Try {assert(false)} catch(err) {Line(err,0)}
return r
}
#=====================================================================================================================
function Line(string s, int ofs) {ofs=default(ofs,0) s=RightStr(s,FindStr(RevStr(s)," ")-1) s=LeftStr(s,StrLen(s)-1) l=int(value(s)+ofs) RT_DebugF("%d",l,name="@line ")}
#=====================================================================================================================

Script may look complicated first but is easy to understand I think. Best I can do today is provide this 'tool' to provoke the failure.
I do assume that the behavior is machine-dependent, since FFT plugins make it more prone to fail,
When reproducing, note that line numbers are logged after the command in that line.

Hi m53!

I've looked at what's happening in detail, and there are three different things going on.


TIVTC.dll cannot handle your clip with a size of 8 pixels. It is crashing in its hand-written assembly routines due to an access violation. This can be solved in your script by increasing the size of your BlankClip to at least 16 pixels.
Average.dll, DePan.dll, and mvtools2.dll are causing access violations all over the place, and multiple times in a single run. They pretty probably don't like the parameters your script is calling them with. This seems irrespective of the clip size they get. Once I've moved them out of my plugins directory (and increased the clip size to 16 for TIVTC), your script executed to the end without errors.
There also seems to be a memory leak that I haven't tracked down yet. This is not what is causing the crashes though, that's going to be the previous two points for almost sure.

Points 1-2 are obviously optimally fixed in the plugins. If no one is going to fix them, your script can at least work around point 1), and as for point 2)... well, you decide what's the best course of action. Note that you might also have additional plugins on your machine that I don't and are causing similar issues.

As for point 3), I'm still investigating it. It might be something inside AviSynth+. But due to your script calling all possible plugin functions and many plugins being buggy (see previous point) by not doing nearly enough parameter validations, this too might as well be a result of an erronous plugin. Even probably.

Anyway, the crash itself wasn't due to AviSynth+.

[martin53]

Ultim,
thank you for the detailed analysis! Especially that you took the time to assign the plugin crashes to the particular plugins. Be sure I estimate this right!
It is still worrying me that the same buggy plugins started from Avisynth 2.6 MT and Avisynth+ make the system as a whole react so differently. And more than the fact it reproduceably crashes after call of certain plugins - which I probably best can solve by explicitly checking in my script and provide a prepared return value - this extreme pausing is weird.
Can Avisynth+ itself do anything more to control/manage plugin crashes even better than now?

[Groucho2004]

Ultim,
thank you for the detailed analysis! Especially that you took the time to assign the plugin crashes to the particular plugins. Be sure I estimate this right!
It is still worrying me that the same buggy plugins started from Avisynth 2.6 MT and Avisynth+ make the system as a whole react so differently. And more than the fact it reproduceably crashes after call of certain plugins - which I probably best can solve by explicitly checking in my script and provide a prepared return value - this extreme pausing is weird.
Can Avisynth+ itself do anything more to control/manage plugin crashes even better than now?
You should also consider that there are DLLs that are needed for some plugins that cause havoc by just loitering in the auto-load directory. Do a search for "AVSFilterNet.dll", for example.

[ultim]

Ultim,
thank you for the detailed analysis! Especially that you took the time to assign the plugin crashes to the particular plugins. Be sure I estimate this right!
It is still worrying me that the same buggy plugins started from Avisynth 2.6 MT and Avisynth+ make the system as a whole react so differently. And more than the fact it reproduceably crashes after call of certain plugins - which I probably best can solve by explicitly checking in my script and provide a prepared return value - this extreme pausing is weird.
Can Avisynth+ itself do anything more to control/manage plugin crashes even better than now?

By the nature of access violations, they will have different effects based on what code is loaded, how much memory was allocated and in what order, what is stored in the memory etc. The point in time where the OS starts screaming about these errors is typically just the tip of the iceberg, and there's a really good chance that memory got corrupted a lot earlier than that. Not every memory corruption manifests itself as an access violation error: when that happens, it merely means - in laymen terms - that the corruption has spread so much that even the OS recognized it. If a video encoding process doesn't crash (which is by pure chance) when an access violation happens in a plugin, that's worse than crashing, because then most probably your encode will be corrupted, and you won't notice it.

All the above means that trying to recover from an access violation is pretty much the worst thing you can do. It might seem odd for a user, but in such a case it really is best practice to just terminate loudly, as soon and fast as you can. Any attempt to recover might make things much worse. Hiding it won't do you any good, because even though you can hide it from the user, it is impossible to negate the corruption it has caused.

My point is: If a plugin is crashing due to an access violation, I'm glad if the whole process crashes. You should be too. I certainly won't try to manage it any "better". I am already performing sub-optimal by turning it into a C++ exception, but that is to have similar behavior to AviSynth 2.6.

[martin53]

Maybe it's not worth to put toooo much work into dealing with touchy plugins.

Why did I write this odd script? Because for writing on the Avisynth+ documentation pages I was just interested in an inventory of all available functions (unrecognized plugin funcs too, out of pure curiosity), and if they were clip or non-clip functions.
Maybe there is a much better way for that than this dumb script?

Of course once the script was in the world, I was also unsettled by the more obvious malfunction in combination with Avisynth+. Yet I agree that malfunctions should either be contained properly, and if they can't - which will sure be mostly the case with access violations - a hidden damage is no better than an obvious one.

[ultim]

Okay, things are starting to shape up. Reported bugs have all been looked at, where appropriate fixed, and MT-functionality has also seen some necessary correctness improvements. I'm thinking about releasing 0.2 pretty soon. What is needed for that at this point is mainly testing, and maybe getting some additional content for the installer into shape. The more adventurous among you can pull the latest test binaries from avs-plus.net/builds (http://avs-plus.net/builds). Starting now, binaries are built with a newer C++ runtime due to the wide availability of VS Community Edition, so if you feel like testing, don't forget to also get the vcredist stuff under the same link. In the official release, the installer will take care of setting up the runtime for you of course.

Depending on what you testers report, the r1779 binaries might be extremely similar to what 0.2 will be. I, for one have been using recent builds in my own production encodes without problems, and the (very few) independent testers I gave them to also had encouraging reports.

As for what the new caching system coupled with the rewritten filters in AviSynth+ are capable of, I'm also showing you these results (http://pastebin.com/7k4KNrAk), based on a real-world and pretty heavy script. And it was running only single-threaded. Now these measurements are not to be generalized, and results will surely depend on both your machine as well your script, but I certainly started smiling when I first saw this comparison. Do you like it? Are you excited for AviSynth+ 0.2?

EDIT: "Quite few" came out wrong upon reading it over. What I mean to say is very few testers, not "quite a few".
EDIT: I forgot to mention, the way how you specify the default MT mode for all filters that don't have it set explicitly has changed. For that, the first argument of SetFilterMTMode now has to be "DEFAULT_MT_MODE" instead of just "" (an empty string). I updated the MT usage notes (http://forum.doom9.org/showthread.php?p=1666371#post1666371), please consult them about how to use MT.

[ultim]

chainik_svp,
you should also try out the latest build linked above. while i cannot stop ffdshow from leaking, this build should have improved the situation a lot. i also briefly looked at ffdshow's avisynth bridge, and i think i saw what the problem is. if you can get me someone who can build ffdshow for a test, i could submit a patch to try and see.

[LoRd_MuldeR]

if you can get me someone who can build ffdshow for a test, i could submit a patch to try and see.

You could try to bother clsid (https://forum.doom9.org/member.php?u=71929) ;)

[chainik_svp]

ultim
Are you excited for AviSynth+ 0.2?

I'd be extremely excited IF the release will contain MT functions.
I mean make MT-enabled version the one and only version for AVS+.


you should also try out the latest build linked above

I will, thanks!


if you can get me someone who can build ffdshow for a test, i could submit a patch to try and see.

yeah, clsid is the one :)

[ultim]

ultim
Are you excited for AviSynth+ 0.2?

I'd be extremely excited IF the release will contain MT functions.
I mean make MT-enabled version the one and only version for AVS+.

The build has all the MT functions. AVS+ won't have non-MT builds anymore. Due to the relatively new nature of MT in Avs+ though, MT-enabled encodes possibly don't yet have the same maturity as non-MT encodes with the same binary. This immaturity is exactly what testing should dissolve over time.

[Groucho2004]

The more adventurous among you can pull the latest test binaries from avs-plus.net/builds (http://avs-plus.net/builds).
One potential problem with that build (x86):
I'm using this script to make avisynth run out of memory:
setmemorymax(3000)
n = 4800
colorbars(width = n * 2, height = n, pixel_type = "yv12").killaudio().assumefps(24000, 1001).trim(0, 999)
SelectRangeEvery(1000, 100, 200)
temporalsoften(4,4,8,15,2)
pointresize(width() - 64, height() - 64).turnleft()
temporalsoften(4,4,8,15,2)
pointresize(width() + 64, height() + 64).turnright()
temporalsoften(4,4,8,15,2)

With the "official" 2.6 I get the usual "We've run out of memory, folks" message after 9 frames.
With AVS+ r1718 I get "Could not allocate video frame. Out of memory." after 4 frames.
AVS+ r1778 does not get past the first frame but also throws "Could not allocate video frame. Out of memory.". However, it also crashes with c0000005 (access violation). Crash dump is here (http://pastebin.com/RNaMwJJD).

[chainik_svp]

In fact I'd LOVE to replace Avisynth with AVS+ in SVP distribution.

We're just waiting for the "official" MT build (32-bit).
And we're all looking forward for the stable 64-bit version.

ultim, if you've any other thoughts about bundling AVS+ with SVP - please contact me :)


AVS+ won't have non-MT builds anymore.

good news, everyone :D

[sl1pkn07]

please keep the non-MT version (the lastest one), for if the linux users are unable to run the MT version

[Groucho2004]

Actually, r1778 crashes with every script. I'm on WinXPSP3.

[chainik_svp]

sl1pkn07

> linux users are unable to run the MT version

Why? o_O

And since AVS+ is by design Windows-only, why won't Linux users switch to native Vapoursynth?

[chainik_svp]

actually, r1778 crashes with every script. I'm on winxpsp3.

vc++ 2013 rt? :)

[Groucho2004]

vc++ 2013 rt? :)
-----Yes.

[innocenat]

sl1pkn07

> linux users are unable to run the MT version

Why? o_O

And since AVS+ is by design Windows-only, why won't Linux users switch to native Vapoursynth?

Wine C++ runtime did not support #include <thread> used by Avs+ MT iirc. But I thought it was already fixed?

[chainik_svp]

may be binaries was built with v120 instead of v120_xp?

> Wine C++ runtime did not support #include <thread> used by Avs+ MT iirc.

so this's the question to WineHQ, right?

and last time I checked, I was able to install any VC runtime i needed
VC runtime isn't provided by Wine o_O

[Groucho2004]

may be binaries was built with v120 instead of v120_xp?
Then it should not run at all (I think).

[sl1pkn07]

sl1pkn07

> linux users are unable to run the MT version

Why? o_O

And since AVS+ is by design Windows-only, why won't Linux users switch to native Vapoursynth?

i mean, maintain the both version, the last one non-MT and the MT version

yes, i use a VS, but for some projects needs maintain AVS system

greetings

[innocenat]

may be binaries was built with v120 instead of v120_xp?

> Wine C++ runtime did not support #include <thread> used by Avs+ MT iirc.

so this's the question to WineHQ, right?

and last time I checked, I was able to install any VC runtime i needed
VC runtime isn't provided by Wine o_O

A year ago (that was when this wine and thread thing was discussed here, and I dont follow wine development much) the situation is that some api arent available so even if you install vcredist it wont run thread at all. And that bug was assigned WontFix in wine bug tracker.

So if that bug is still present, user kinda require ST version.

[qyot27]

The issues with Wine are gone (http://forum.doom9.org/showthread.php?p=1707414#post1707414) so long as you're using the latest Wine 1.7, although the native Wine dlls for msvc[p|r]120 are still insufficient, so you need the MS ones (since 1.7 is the -devel branch, this could very well change before 1.8). My various personal builds of AviSynth+ from r1718 onwards seem to work fine - I get similar speeds as I was with 2.6 MT (might be mostly due to getting bottlenecked in HCenc). And I was using Prefetch(4) and setting the filter mode with the .avsi, so the MT should be active.

The only problem is that the MSVC 2012 and 2013 runtime installers error out. The DLLs work, the installers don't. So trying to use the redist bundled in AviSynth+'s installer won't work on Wine. You have to extract the dlls from the MS redists and copy them into Wine's system32 directory, and then add overrides for those DLLs in winecfg. This was true on both OSX 10.10 with Homebrew and Ubuntu 14.10 with the ubuntu-wine PPA. Really, winetricks is the place to handle that part, and it should be only a matter of time before vcrun2012/vcrun2013/etc. options get added to it (if they haven't been already).

And since AVS+ is by design Windows-only,
Classic AviSynth has that limitation; there is work being done to resolve that issue in AviSynth+ (it's just taking a while).

[ultim]

One potential problem with that build (x86):
I'm using this script to make avisynth run out of memory:
setmemorymax(3000)
n = 4800
colorbars(width = n * 2, height = n, pixel_type = "yv12").killaudio().assumefps(24000, 1001).trim(0, 999)
SelectRangeEvery(1000, 100, 200)
temporalsoften(4,4,8,15,2)
pointresize(width() - 64, height() - 64).turnleft()
temporalsoften(4,4,8,15,2)
pointresize(width() + 64, height() + 64).turnright()
temporalsoften(4,4,8,15,2)
...

Yeah this script runs out of memory. I'll try to adapt some things to shrink the caches more aggressively under extreme memory pressure.

As for XP-compatibility, all things are compiled with the xp-toolset, so at least from that aspect, it should be fine. I wonder if there was some other oversight on my part (e.g. using an unsupported API or dunno what), I admit I haven't tested on XP at all. Will look into it.

[Boulder]

It's been a year since I said "I'm going to hop on from Avisynth MT (SEt's latest v2.6 build) to Avisynth+ just for some testing." and now it seems that it is the time if you feel that the multithreading issues should be gone. Do you have anything specific you wish to have tested? Should the output of Avisynth 2.6 MT and Avisynth+ be identical? And the most important question: is MT enabled automatically or does it need to be set to achieve maximum performance?

[chainik_svp]

MT is not working for me in the latest build o_O
The very same script

global threads=9
SetFilterMTMode("",2)
SetFilterMTMode("ffdShow_source",3)
ffdShow_source()
...
Prefetch(threads)
is working as expected with October build and working with a 1-thread speed with the latest one

[ultim]

and now it seems that it is the time if you feel that the multithreading issues should be gone.
At least the ones that I know of. Whether there is anything else undiscovered, we will find out as more and more people test. So yes, now is the right time to test again.

Do you have anything specific you wish to have tested?
Nothing really in specific. Just use your regular scripts as you would normally do, and let me know if you see any problems. Plus-points for anybody who also tries out their script(s) using MT, but for that be sure that the correct filter modes have been specified. I cannot stress this enough: you can only expect MT-mode to behave sanely if you have the correct MT mode setting for all filters in your script.
Here is a start for MT mode settings: https://pad.riseup.net/p/avs_plus_mt_modes . Never-fading fame for all who make fixes or additions to that list.

Should the output of Avisynth 2.6 MT and Avisynth+ be identical?
The output should be visually indifferentiable, but not bit-exact identical.

And the most important question: is MT enabled automatically or does it need to be set to achieve maximum performance?
It needs to be set up in your script. Not just for maximum performance, but also for correct operation in general. The setup is composed of two parts: 1) Setting the MT modes in the beginning, then 2) enabling MT by adding the Prefetch-line to the end of your script. For detailed instructions (including example) about using MT, see this post (http://forum.doom9.org/showthread.php?p=1666371#post1666371).

[ultim]

MT is not working for me in the latest build o_O
The very same script

global threads=9
SetFilterMTMode("",2)
SetFilterMTMode("ffdShow_source",3)
ffdShow_source()
...
Prefetch(threads)
is working as expected with October build and working with a 1-thread speed with the latest one

Hi,
Replace
SetFilterMTMode("",2)
by
SetFilterMTMode("DEFAULT_MT_MODE",2)

Sorry I forgot to mention earlier.

[Boulder]

@ultim: Thanks a lot, I'll try to find the time to experiment this week :)

[Groucho2004]

Yeah this script runs out of memory. I'll try to adapt some things to shrink the caches more aggressively under extreme memory pressure.

As for XP-compatibility, all things are compiled with the xp-toolset, so at least from that aspect, it should be fine. I wonder if there was some other oversight on my part (e.g. using an unsupported API or dunno what), I admit I haven't tested on XP at all. Will look into it.
As I mentioned later, r1778 crashes with every script on my WinXP.

[burfadel]

As I mentioned later, r1778 crashes with every script on my WinXP.

Windows XP.

Compiling for Windows XP and making it universal is detrimental to users of Windows 8.1 etc.

[chainik_svp]

SetFilterMTMode("DEFAULT_MT_MODE",2)

Yeah, thanks.

I see some very strange behavior when playing video with SVP.
Everything looks great, except when I set number of threads to 6 or 11 video playback becomes choppy.
Every other number (tested values from 2 to 15) gives smooth playback.
I'm on 2 cores / 4 threads SandyBridge now. Will test it with my FX-8320 later...

===
it looks like source filter (which is ffdshow_source()) became inaccurate with those values
i.e. the next filter in the chain (SVSuper()) sometimes receives the same frame when it asks for N and N-1
so motions vectors are zero and playback is choppy

most likely this's some issue with the cache

[Groucho2004]

Windows XP.

Compiling for Windows XP and making it universal is detrimental to users of Windows 8.1 etc.
- Can you share any examples from your development experience that would support such a generalized statement?
- Do you work for Microsoft?
- Just trolling?

[ultim]

it looks like source filter (which is ffdshow_source()) became inaccurate with those values
i.e. the next filter in the chain (SVSuper()) sometimes receives the same frame when it asks for N and N-1
so motions vectors are zero and playback is choppy

most likely this's some issue with the cache

Hmmm, I'm not sure about that. Might as well be just DirectShow messing up, as it is known to be not frame-accurate during seeks in general. This is often not a problem during sequential processing, but with MT-mode enabled, directshow will recieve out-of-order frame requests even during sequential playback.

Edit: Consideirng that in your case avisynth is not running as a standalone directshow filter but as part of ffdshow, it might actually be frame-accurate depending on which source filters ffdshow chooses to use. I don't know enough about ffdshow internals to judge that tbh.

[chainik_svp]

I never saw this before with Avisynth ;)
And again, in AVS+ the issue is only with 6 or 11 threads. Some kind of magic numbers o_O

[ultim]

Windows XP.

Compiling for Windows XP and making it universal is detrimental to users of Windows 8.1 etc.

Don't worry burfadel, support for XP in AviSynth(+) does not have any negative effects that I can think of on post-XP Windows users.

[mark0077]

SetFilterMTMode("DEFAULT_MT_MODE",2)

Yeah, thanks.

I see some very strange behavior when playing video with SVP.
Everything looks great, except when I set number of threads to 6 or 11 video playback becomes choppy.
Every other number (tested values from 2 to 15) gives smooth playback.
I'm on 2 cores / 4 threads SandyBridge now. Will test it with my FX-8320 later...

===
it looks like source filter (which is ffdshow_source()) became inaccurate with those values
i.e. the next filter in the chain (SVSuper()) sometimes receives the same frame when it asks for N and N-1
so motions vectors are zero and playback is choppy

most likely this's some issue with the cache

With normal Avisynth MT I always have strange issues with audio going out of sync depending on what number of buffer aheads I use. See my previous posts at the thread below. The issue is still there for me, I again have to pick some "magic" good numbers of buffer ahead like 14 or 15 to stop audio going out of sync. (I'll try today with avisynth+ to see can I get audio to go out of sync).

http://www.svp-team.com/forum/viewtopic.php?id=1011&p=3

I definitely feel there's some work needed in the buffer ahead areas to stop choppyness and also audio going out of sync. I'm not sure whats the best way to illustrate this so that a solution could be found.

[ultim]

Hmmm, I'm not sure about that. Might as well be just DirectShow messing up, as it is known to be not frame-accurate during seeks in general. This is often not a problem during sequential processing, but with MT-mode enabled, directshow will recieve out-of-order frame requests even during sequential playback.

Another possible cause of choppy real-time playback is when threads gang up (e.g. their results arrive in "waves"). In that case they still deliver the same performance on average, but not evenly distributed in every short time window. To see if this is the case, you'd have to encode a piece of video to file with all the processing, then play back the pre-processed file without avisynth. If the playback is only choppy during real-time processing but not in the encoded file itself, this is exactly what is happening.

[mark0077]

I am having problems with audio desync when switching to avisynth+ (r1778) with SVP. I assume its down to the issue with ffdshow buffer ahead weirdness? Can anyone suggest a good number of avisynth+ threads, SVP cores, and ffdshow buffer aheads to try? I'm using core i7 920.

I was previously using SETs lates Avisynth MT with its threads, SVP cores and ffdshow buffer ahead set to a value of 15 where all worked fine. So far I have tried lowering the Avisynth+ threads down to 8 / 6 / 4 but audio seems really badly out of sync AND/OR video being choppy in all combinations I have tried.

[chainik_svp]

utim

This is not about timing, frames are rendering uniformly.
I can visualize motion vectors, and it's clear that they're all zeroes at some frames.
Zero-sized MVs mean source frames used for motion estimation was identical.

But I still can't reproduce this effect via avs2avi or figure out any simple test scenario for ffdshow o_O

May be you want to install SVP? :)

[ultim]

But I still can't reproduce this effect via avs2avi or figure out any simple test scenario for ffdshow o_O
Does that mean it only happens with ffdshow?

[burfadel]

Don't worry burfadel, support for XP in AviSynth(+) does not have any negative effects that I can think of on post-XP Windows users.

Ah ok. Still, I don't believe support should be in place for XP any more, it only encourages people to stick with it longer.

[chainik_svp]

I've checked frames returned by ffdshow_source() by inserting some debug code into Cache::GetFrame that computes 'checksum' of the Y plane.

Normal behavior (num of threads !=6): lookup always returns LRU_LOOKUP_NOT_FOUND, and for each 'n' returned frame is a new one (with contents different from frame 'n-1').

81: .... 8B35E6F
82: .... 8A80C08
83: .... 899A3CA
84: .... 89D3DED
85: .... 8B1A39A

Invalid behavior (num of threads ==6): every other frame is equal to the previous one.

81: .... 8B35E6F
82: .... 8B35E6F
83: .... 899A3CA
84: .... 899A3CA
85: .... 8B1A39A
86: .... 8B1A39A

[ultim]

I've checked frames returned by ffdshow_source() by inserting some debug code into Cache::GetFrame that computes 'checksum' of the Y plane.

Normal behavior (num of threads !=6): lookup always returns LRU_LOOKUP_NOT_FOUND, and for each 'n' returned frame is a new one (with contents different from frame 'n-1').

81: .... 8B35E6F
82: .... 8A80C08
83: .... 899A3CA
84: .... 89D3DED
85: .... 8B1A39A

Invalid behavior (num of threads ==6): every other frame is equal to the previous one.

81: .... 8B35E6F
82: .... 8B35E6F
83: .... 899A3CA
84: .... 899A3CA
85: .... 8B1A39A
86: .... 8B1A39A

I'll see if I can reproduce this.

[chainik_svp]

The very same magic numbers (6 and 11) on 4 cores / 8 threads AMD FX-8320