Plugin autoloading

[dividee]

If nobody is working on it yet, I'd like to implement that.
Here is how I envision it:
When a Script Environment is created, it looks up a specific value in the registry: the plugin directory location. It then loads every plugin DLL and import every avs it founds there.
We'll see if that cause any slowdown or a noticeably bigger memory usage; in that case a mechanism to unload unused plugins could be necessary.

The plugin directory location should be set during installation. Does someone wants to work on an installer ?

[Schultz]

I can create a AviSynth installer using NSIS if you would like.. (Actually i have already created one long ago having Avisynth 1.05 for my friends but didn't update it since then) you can look it up at
http://www.edensrising.com/avisynthsetup
i can do it or if you want i can send the script to anyone that wants it..

[sh0dan]

@dividee: Sounds like a good solution - if there are any performance problems, they should proably be fixed, instead of hacking around it. Should the installer ask for a plugin directory, or should it just be created in the Avisynth installation directory?
How about name conflicts. I don't think the current solution is a good one - could it be possible to activate the newest DLL?

@Schultz: The best solution would probably be for you to be added to the SourceForge project - would that make sense?

[Richard Berg]

That's exactly how I envisioned it, dividee.

I have in the not too distant past authored an MSI database from scratch that does exactly what we need. Unfortunately compiling it requires tools I'm not sure I can share. The good news being that I know the Windows Installer platform inside & out; I could probably make a decent package using Orca if that says anything :eek:

On the coding front, I've played with VDub's auto-loading code some. (Search for "int FilterAutoloadModules"). We require a bit more work (3 types of plugins), but still only looking at <100 lines of code in all likelihood. Just gotta remember to change the priority to favor plugins over interals.

Edit: NSIS isn't Windows Installer based :( Suppose it's no surprise these guys haven't caught up to 1999 tech yet; it took them 4 years to make a working shuffle function for chrissake.

[Schultz]

Well i know Justin who created NSIS wants a simple nice efficient installer with minimal size overhead.. i think that current version 1.98 only has 34k for the Header size of the installer and the rest is for compressed data.. Plus its opensource to modify the looks or what not to how you want.. i know some people have expanded the project alows you to totally change the UI http://sourceforge.net/projects/nsis2k/

but if dividee would like yea i can send him the script for him to put it on SourceForge.

[trbarry]

Can we have it so any filters explicitly loaded in the avs script take preference. In testing I often switch back and forth from debug to release versions, etc. Just date checking wouldn't do it and would often be confusing.

And I don't think there would be too much performance hit if Avisynth was only doing LoadPlugin.

- Tom

[Richard Berg]

Actually, AFAICT, functions only get loaded into the (backwards) linked list when the parser hits them. Nevertheless, I've bumped the limit on plugins to 50 from 30 in anticipation of large autoloading directories (I know my copy of VDub loads ~50 plugins on startup, and AVS is much more powerful!)

Tom -- you mean if plugins have the same name but are located in different directories. I don't know exactly how that's handled at the moment, but it may require changing the anti-conflict code since the autoloaded versions will certainly get loaded before ones in the script even with the recent change in ordering in avisynth.cpp.

[poptones]

Comparing this to veedub is rather too simplistic. Consider veedub doesn't actually DO anything with those filters until they are used. You have to manually select each one and then specify parameters - just like we do in avisynth. All veedub "imports" is a list of available filters - the namespaces are unique to each one. That is very differnt than veedub, where I can have three identical filters separated only by rev level, and each can exist without conflict.

Consider:

In the "plugins" directory exists one script called "plugins.avs." This is simply a list of plugins to be imported using the already existing standard terminology - it's just an avisynth script.

When a script is invoked "plugins.avs" is automatically loaded.

This would allow a "type manager" type interface. Default behavior could be simply to add new filters to the list when they are detected, but you could change that default behavior with a registry key. So when someone builds a "plugin manager" interface it can change the key and, thus, the behavior.

Power users get power, newbs get simplicity. All it takes for either one is the installation of a new component. And since AVISynth now has an actual home, those components will be a LOT simpler to add.

[trbarry]

Richard -

I thought I remembered that Ben RG wrote the most recently loaded one was used. But I just went looking for the reference and couldn't find it so maybe I was imagining things. And I have not tested it.

But usually it is a good idea to have explicitly requested stuff override the defaults.

- Tom

edit: Of course I could obviously just delete anything from a plugin directory that I was testing, so it's not the end of the world either way. But it would be nicer.

[Schultz]

If you want i just uploaded a zip file into the site above that has all the nsi script and all the Dir structure stuff.. Its uses Nsis2.0a6 from the link i gave above.. You can use it if you like.

[dividee]

Sorry for the long silence but I was pretty busy yesterday.
I made some tests and I have to admit I was wrong! I'm sorry about the confusion I have caused. Avisynth already prioritize scope in a rather sensible manner:
* the lastly loaded plugin supercedes previous plugins and built-in filters
* script functions supercedes built-in filters but NOT loaded plugins

You can even do things like that:

# foo.dll and foo2.dll exports a foo(clip) filter
LoadPlugin("foo.dll")

function bar(clip c)
{
LoadPlugin("foo2.dll")
foo(c) # call the function in foo2.dll
}

#insert source filter
foo # call the filter in foo.dll



It means I'll have much less work :)
I have a functionnal AutoLoadPlugin function which loads all DLL and AVS in the plugin directory. It loads all DLLs before any AVS, but doesn't load them in a specific order. I don't think this really a problem, tell me otherwise.

It's also not a problem if some AVS loads some plugins, as avisynth check if a plugin is already loaded. However, if some AVS Import other ones, it will cause the imported ones to be loaded multiple times. This is not too bad since it will work but consumes some ressources.

[edit:] I just saw Richard's mods to avisynth.cpp to give priotiry to plugins. Thanks, but shouldn't script functions still have priority over plugins?

@Schultz
I like your installer (except it does install avisynth in the Temp folder in lieu of the system dir ;) ). If everyone aggree I'll make a stripped down version which only install avisynth and the docs, since that's all we have in the CVS right now. Or maybe Richard will make one that is Windows Installer based ? ;)

[Schultz]

Originally posted by dividee

@Schultz
I like your installer (except it does install avisynth in the Temp folder in lieu of the system dir ;) ). If everyone aggree I'll make a stripped down version which only install avisynth and the docs, since that's all we have in the CVS right now. Or maybe Richard will make one that is Windows Installer based ? ;)

sorry was playing around with the installer adding stuff and didn't realize that it was placing it into the $TEMP directory insted of $SYSDIR in NSIS>. just place
SetOutPath $SYSDIR right before the File avisynth\avisynth.dll which you probalby already knew.. Or updated zip file and script is on the server..

[Richard Berg]

I just saw Richard's mods to avisynth.cpp to give priotiry to plugins. Thanks, but shouldn't script functions still have priority over plugins?
If anyone is able to improve on the built-in filters it would be nice of course if they sent the source to us, but I don't think it should be required.
Or maybe Richard will make one that is Windows Installer based ?
I could whip up some XML in about 5 minutes, but I'd never feel right using internal tools and the GPL project (http://msi2xml.sourceforge.net/) has a LONG way to go to be useful. MSI gives you lots of stuff for free (repair/uninstall/versioning/etc.), but most of the advanced features we can do without unless since we're not designing for a corporate rollout...plus, Schultz is already doing the work for us :) I'll stick the script on the CVS at some point.

[dividee]

In fact, the ordering you changed has nothing to do with the priority of plugins vs script functions vs built-ins functions (I was abused too).
The priority is defined by the orders in which functions are inserted in the Function Table (which stores functions in a linked list and works in a LIFO way) and the order of scanning in FunctionTable::Lookup (external functions are searched before built-ins).
I now implemented "prescanning" instead of "preloading": All plugins in the plugin directory are loaded, the functions are read and the plugins are unloaded. It then loads the plugins in memory as soon as a Lookup is performed on one the function it exports.

It was quite some work before I found a not-too-trashy implementation (I've started at least four of them, one was even complete and worked, but involved some dynamic_cast (yuk!) ). Still, I wouln't say it's the most elegant piece of code in the world, but given the constraint that the plugin interface can't be changed it's the best one I found. Even without that, the memory usage seems to increase only by approximately the size of the dll (when loaded and unused). As the average plugin is less than 100K, having 30 plugins there would only increase memory usage by less than 3MB.
But at first I was thinking it would be easier, and once I started, I didn't want to give up ;)

I'll commit my stuff soon, I just want to test it a little more. But I don't think we should rush to another release (we are not even sure if the new AVISource is buggy or not).

[dividee]

I just commited the auto-loading functionnality. It compiles cleanly under VS.NET, I hope it's ok with VS6.
To use it, you'll need to manually create a String Value HKEY_LOCAL_MACHINE\Software\Avisynth\PluginDir in the registry with the name of the plugin directory, or use Schultz installer. Then put all the plugins & "library scripts" you have in that dir :)
Please consider this functionnality alpha...

[ARDA]

I've compiled with VS6 without any problem, just two warnings that
I've had before your autoplugin commit. Just made a test with
my usual scripts, nothing wrong; didn't do anything with auto-loading functionnality.

C:\copydvd\avisynth\avisynth_204_src\Copia de avisynth_204_src\VD_Audio.cpp(1448) :
warning C4390: ';' : empty controlled statement found; is this the intent?

LINK : warning LNK4089: all references to "WINMM.dll" discarded by /OPT:REF

Are these warning importants? I've detected nothing with my usual
scripts, or am I losing something ?

Thanks Arda

[dividee]

Yes, these warning only appears in Release builds, not in Debug builds. They are both harmless, although I don't quite understand the second one: it says the mentionned library wasn't used and so has been discarded; but if you remove that library from the library modules it doesn't link :confused:

[Richard Berg]

I remember investigating that link warning when you first patched in the VDub audio; I could only conclude it was either a really badly-worded message, or a linker bug (always a distinct possibility with VS6).

[dividee]

BTW I don't get this warning in VS.NET (although I get a lot of "signed/unsigned mismatch" and "conversion from xxx to yyy, possible loss of data", at the same warning level, from the compiler).
It might indeed be a bug in the VS6 linker.