Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Closed Thread
 
Thread Tools Search this Thread Display Modes
Old 1st October 2013, 09:26   #61  |  Link
hanfrunz
Registered User
 
hanfrunz's Avatar
 
Join Date: Feb 2002
Location: Germany
Posts: 540
What avisynth development really needs is a meeting of some of the core developers in person. I have no idea if this is possible, but an avisynth developer conference/camp would be cool. Maybe you find another conference, where you could meet. For example the 30c3 in Germany this december, etc.
hanfrunz is offline  
Old 1st October 2013, 11:26   #62  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
Quote:
Originally Posted by hanfrunz View Post
What avisynth development really needs is a meeting of some of the core developers in person.
As far as I know, IanB is The Last Man Standing from that group.
Groucho2004 is offline  
Old 1st October 2013, 21:22   #63  |  Link
ultim
AVS+ Dev
 
ultim's Avatar
 
Join Date: Aug 2013
Posts: 359
Yuhuuuu, new build, and new code on GitHub!

So, what has changed:
- First of all, the crash-on-out-of-memory bug is hopefully fixed. Should be.
- There is a brand new plugin-system in place, and if you work with scripts that use a lot of plugins, you should notice that they load faster.
- You can have multiple plugin directories. Exact semantics in my next post.
- LoadCPlugin (or Load_Stdcall_Plugin) is now a synonym for LoadPlugin. LoadPlugin will load C-plugins and LoadCPlugin will load normal plugins. They are one and the same. No difference.
- Hence, C plugins are also autoloaded.
- LoadVFAPIPlugin() is out of order for now. I'm not planning on removing it, I just need some info how to correct it.

Changes noteworthy for developers:
- Invoke finally stops throwing exceptions as a "normal condition" -> better debuggability
- VFAPI and VirtualDub filter loading are now separated into their own plugins, and are not in core any more.
ultim is offline  
Old 1st October 2013, 21:42   #64  |  Link
ultim
AVS+ Dev
 
ultim's Avatar
 
Join Date: Aug 2013
Posts: 359
Okay, so how do multiple plugin directories interact with plugin autoloading?

As a recap, here is how it used to work in the official Avisynth:
- Look for the string HKEY_CURRENT_USER/Software/Avisynth/PluginDir2_5 in the registry. If it exists, load plugins from the path specified there and stop.
- If the above string didn't exist, look in HKEY_LOCAL_MACHINE/Software/Avisynth/PluginDir2_5. Try to load plugins from the path specified there.
- Done.

First thing to note, is that classic Avisynth only ever searches for plugins in one single directory. It only knows two directories (both specified in the registry), and it only tries the second path if there is no entry for the first one.

Avisynth+'s autoloader has a list of autoload directories. It iterates over all those directories and tries to load all plugins from each. But (and a big but!) it will not load a plugin from a directory if another plugin with the same basename is already loaded. The basename of a plugin is simply its file name without the extension.

The expected use case is that you can now overlay a new plugin directory on top of another one. Avisynth+ then would load all plugins from the first folder, then load only those plugins from the second that weren't loaded from the first, then those from the third that weren't loaded from the first or second and so on. For example, let's say your usual plugin folder has a lot of plugins you normally use. But at one time you have a small number of updated plugins that you only want to use from a few scripts, but you do not yet want to replace your existing plugins globally. Then you'd just add a new plugin overlay folder, with only the new plugins in it, and that's it. All scripts that specify the new folder will autoload all plugins from your usual one, except for the new plugins, which would get loaded from the new folder. All your other scripts will still use your old plugins.

My next post will tell you how to add and remove autoload folders.
ultim is offline  
Old 1st October 2013, 21:51   #65  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
Quote:
Originally Posted by ultim View Post
Avisynth+'s autoloader has a list of autoload directories. It iterates over all those directories and tries to load all plugins from each. But (and a big but!) it will not load a plugin from a directory if another plugin with the same basename is already loaded. The basename of a plugin is simply its file name without the extension.
Might be a bad idea. Take for example the RemoveGrain package. It has several DLLs, each built with different optimizations (SSE, SSE2, etc.). They are named "RemoveGrainSSE.dll", "RemoveGrainSSE2.dll", you get the idea. Yet, they all have the same name in AvisynthPluginInit. I bet that 3 out of 4 users dump the lot in their autoload directory.
Groucho2004 is offline  
Old 1st October 2013, 21:52   #66  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
Quote:
Originally Posted by ultim View Post
First of all, the crash-on-out-of-memory bug is hopefully fixed. Should be.
Yes, it is.
Groucho2004 is offline  
Old 1st October 2013, 22:09   #67  |  Link
ultim
AVS+ Dev
 
ultim's Avatar
 
Join Date: Aug 2013
Posts: 359
By default, Avisynth+'s autoload folder list has four paths in it, in this order:
- PluginDir+ in Software/Avisynth in HKEY_CURRENT_USER
- PluginDir+ in Software/Avisynth in HKEY_LOCAL_MACHINE
- PluginDir2_5 in Software/Avisynth in HKEY_CURRENT_USER
- PluginDir2_5 in Software/Avisynth in HKEY_LOCAL_MACHINE

This means, if there are ever plugins which will only work with Avs+ but not with classic Avs, you can put them into one of the "PluginDir+" folders. Avisynth+ will then use the classic plugins from the normal Avisynth, but if there are versions of some plugins written for Avisynth+, it will use them instead, and the classic avisynth.dll will still not be bothered with them. This is all without you having to lift a finger (except for adding the "PluginDir+" values to the registry once, until we have an installer). So to summarize all this, you have the ability to define a plugin autoload folder in the registry which will only be used by Avs+, but not by Avs, in addition to your classic plugins.

However, another new functionality offered by Avisynth+, is that now you can also specify autoload paths in the scripts. There are two functions for this:

- AddAutoloadDir(string path, bool toFront): this will add a new autoload folder. The string parameter is obligatory, it is the folder path where to load from. The second boolean parameter is optional, and if true (default), it will add the path to the front/beginning of the autoloader's list, which means it will be searched earlier than the rest. If it is false, the path will get added to the end of the list, so it will get searched last (unless you again add another one to the end).
- ClearAutoloadDirs(): This will clear all the paths from the autoloader's list. Note that it is NOT a reset to the default state. ClearAutoloadDirs() will clear all folders, so if you don't add new ones after that, you have disabled the autoload functionality. This is, BTW, also a way to disable autoloading for a particular script in Avisynth+.

Here's an important note: You can only call these functions if no plugin has been autoloaded yet. Autoloading happens if the first unknown function is looked up. This means you can only call AddAutoloadDir or ClearAutoloadDirs if you have only made calls to built-in functions up to that point in the script. I suggest you start your scripts with these calls to avoid any problems.

There is only one thing left to discuss: Are there any special directories you can reference from your script? You bet there are
- SCRIPTDIR is the folder of the most current script. It is the path of the imported script if your script calls import()
- MAINSCRIPTDIR is the folder of your main script, the one where execution started
- PROGRAMDIR is the folder of the executable running the current script
- USER_PLUS_PLUGINS is the string stored in PluginDir+ in Software/Avisynth in HKEY_CURRENT_USER
- MACHINE_PLUS_PLUGINS is the string stored in PluginDir+ in Software/Avisynth in HKEY_LOCAL_MACHINE
- USER_CLASSIC_PLUGINS is the string stored in PluginDir2_5 in Software/Avisynth in HKEY_CURRENT_USER
- MACHINE_CLASSIC_PLUGINS is the string stored in PluginDir2_5 in Software/Avisynth in HKEY_LOCAL_MACHINE
... all these special constants are case-sensitive for now.

Examples
- If you want plugins to be autoloaded from the script's "autoload" directory too, you'd write:
AddAutoloadDir("MAINSCRIPTDIR/autoload")

- If you want plugins to be autoloaded from the script's "autoload" directory, only from there and nowhere else, you'd write:
ClearAutoloadDirs()
AddAutoloadDir("MAINSCRIPTDIR/autoload")

- If you wanted to manually recreate the default state of the autoloading folder list, you'd write:
ClearAutoloadDirs()
AddAutoloadDir("USER_PLUS_PLUGINS", false)
AddAutoloadDir("MACHINE_PLUS_PLUGINS", false)
AddAutoloadDir("USER_CLASSIC_PLUGINS", false)
AddAutoloadDir("MACHINE_CLASSIC_PLUGINS", false)

Last edited by ultim; 1st October 2013 at 22:37. Reason: Corrected example
ultim is offline  
Old 1st October 2013, 22:30   #68  |  Link
ultim
AVS+ Dev
 
ultim's Avatar
 
Join Date: Aug 2013
Posts: 359
Quote:
Originally Posted by Groucho2004 View Post
Might be a bad idea. Take for example the RemoveGrain package. It has several DLLs, each built with different optimizations (SSE, SSE2, etc.). They are named "RemoveGrainSSE.dll", "RemoveGrainSSE2.dll", you get the idea. Yet, they all have the same name in AvisynthPluginInit. I bet that 3 out of 4 users dump the lot in their autoload directory.
Yes, in this case they'd be "recognized" as separate plugins. But even if Avs+ could tell that they are different versions of the same plugin, it still wouldn't be able to decide which one to load (for that it'd neeed some mechanism to determine the used optimizations in a binary, which is not really feasable). So the choice of a different basename wouldn't really help here anyway, AFAICT. The problem is really that RemoveGrain doesn't use dynamic dispatching of routines based on the current machine's instruction set.
ultim is offline  
Old 1st October 2013, 22:52   #69  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
Quote:
Originally Posted by ultim View Post
The problem is really that RemoveGrain doesn't use dynamic dispatching of routines based on the current machine's instruction set.
Removegrain was just an example and yes, the missing CPU dispatcher is the problem.
Another thing is to improve somehow the detection of garbage in the autoload directories (i.e. old plugins which are incompatible with 2.6). However, querying the interface version for each plugin every time would probably be too time consuming.
Groucho2004 is offline  
Old 2nd October 2013, 00:07   #70  |  Link
qyot27
...?
 
qyot27's Avatar
 
Join Date: Nov 2005
Location: Florida
Posts: 1,419
What about an index that caches the necessary functions out of the plugins (maybe something like what fontconfig does, where the cache is only regenerated when it detects a change in the directory contents)? Would that allow more flexibility to determine which version of a plugin should be used or which plugins actually get autoloaded? Checking a cache file would probably be a lot faster than checking the plugins themselves, right?
qyot27 is offline  
Old 2nd October 2013, 00:37   #71  |  Link
Gavino
Avisynth language lover
 
Join Date: Dec 2007
Location: Spain
Posts: 3,431
Quote:
Originally Posted by ultim View Post
- If you want plugins to be autoloaded from the script's "autoload" directory too, you'd write:
AddAutoloadDir("MAINSCRIPTDIR/autoload")
In what contexts do MAINSCRIPTDIR and the other 'special' names get replaced with the corresponding folders?
In all strings, or only when used in the argument to AddAutoloadDir?
__________________
GScript and GRunT - complex Avisynth scripting made easier
Gavino is offline  
Old 2nd October 2013, 02:00   #72  |  Link
qyot27
...?
 
qyot27's Avatar
 
Join Date: Nov 2005
Location: Florida
Posts: 1,419
On the C-plugin autoloading front, I can verify that AssRender now autoloads, but FFMS2's C-plugin (github.com/FFMS/ffms2/commits/c_plugin) does not, even when I try to load it with LoadPlugin.

On a guess, I think it *might* have to do with FFMS2 using the same modified-for-2.6-colorspaces version of avisynth_c.h that x264 and FFmpeg do. So a feature request would be to merge in those changes and finally bring avisynth_c.h up to speed.

Last edited by qyot27; 2nd October 2013 at 02:03.
qyot27 is offline  
Old 2nd October 2013, 06:46   #73  |  Link
qyot27
...?
 
qyot27's Avatar
 
Join Date: Nov 2005
Location: Florida
Posts: 1,419
The build instructions I posted earlier have been revamped and fleshed out. It now includes DirectShowSource setup.
qyot27 is offline  
Old 2nd October 2013, 08:27   #74  |  Link
ultim
AVS+ Dev
 
ultim's Avatar
 
Join Date: Aug 2013
Posts: 359
Quote:
Originally Posted by Groucho2004 View Post
Another thing is to improve somehow the detection of garbage in the autoload directories (i.e. old plugins which are incompatible with 2.6). However, querying the interface version for each plugin every time would probably be too time consuming.
I'm not sure if that is what you mean, but both Avs and Avs+ already query interface versions. They try to load the 2.6 interface from a plugin first, and if that is not supported, they try to load the 2.5 interface. Avs+ also tries to load the C interface if both of the previous ones fail. In the future, the C interface should probably be prioritized over 2.5.

Quote:
Originally Posted by qyot27 View Post
What about an index that caches the necessary functions out of the plugins (maybe something like what fontconfig does, where the cache is only regenerated when it detects a change in the directory contents)? Would that allow more flexibility to determine which version of a plugin should be used or which plugins actually get autoloaded? Checking a cache file would probably be a lot faster than checking the plugins themselves, right?
I'm not sure how that would help. Even if it detected a plugin was changed/added/removed, that still doesn't tell us much about whether it is the "correct" version. Can you elaborate on your idea further?

As a side note, because Avs(+) doesn't have a continuously running process in the background, plugin change detection cannot be implemented efficiently. Even fontconfig has to be notified from an external application if the font folder changes. And purely for performance, implementing a plugin cache isn't needed for Avisynth. As a comparison, there are much fewer plugins than fonts, and the plugins are also much smaller.

Quote:
Originally Posted by Gavino View Post
In what contexts do MAINSCRIPTDIR and the other 'special' names get replaced with the corresponding folders?
In all strings, or only when used in the argument to AddAutoloadDir?
Only in AddAutoloadDir(), and even there, only if they are at the very beginning of the string. These get replaced to absolute folder paths, so if they are not at the beginning of the string, replacing them would only result in an invalid path (e.g. you'd end up with "c:" in the middle of your path).

Quote:
Originally Posted by qyot27 View Post
On the C-plugin autoloading front, I can verify that AssRender now autoloads, but FFMS2's C-plugin (github.com/FFMS/ffms2/commits/c_plugin) does not, even when I try to load it with LoadPlugin.

On a guess, I think it *might* have to do with FFMS2 using the same modified-for-2.6-colorspaces version of avisynth_c.h that x264 and FFmpeg do. So a feature request would be to merge in those changes and finally bring avisynth_c.h up to speed.
Nice spotting. I'll have a look at it when I get home.

Quote:
Originally Posted by qyot27 View Post
The build instructions I posted earlier have been revamped and fleshed out. It now includes DirectShowSource setup.
Thx.

Last edited by ultim; 2nd October 2013 at 08:50. Reason: Removed paragraph about function call matching, as I need to verify that once more.
ultim is offline  
Old 2nd October 2013, 21:40   #75  |  Link
qyot27
...?
 
qyot27's Avatar
 
Join Date: Nov 2005
Location: Florida
Posts: 1,419
Quote:
Originally Posted by ultim View Post
I'm not sure how that would help. Even if it detected a plugin was changed/added/removed, that still doesn't tell us much about whether it is the "correct" version. Can you elaborate on your idea further?

As a side note, because Avs(+) doesn't have a continuously running process in the background, plugin change detection cannot be implemented efficiently. Even fontconfig has to be notified from an external application if the font folder changes. And purely for performance, implementing a plugin cache isn't needed for Avisynth. As a comparison, there are much fewer plugins than fonts, and the plugins are also much smaller.
The original idea actually was rooted in how I understood the original AviSynth's plugin autoloading to work, where all plugins in the autoload folder were loaded so it could identify where functions were, then everything was unloaded, and finally the necessary ones were loaded again. That initial step could cause issues if a plugin was prone to crashing, even if said plugin wasn't even used in the script (one of the variants of WarpSharp was known to cause issues like this).

The cache idea was mostly thought up to avoid that scenario, since it would store the names and plugin locations of the functions and avoid having to perform those first two steps and any instability that could cause - instead, it'd read the cache, compare it to the script, and then load only the plugins used in the script.

But if there's an issue with plugin name collisions, a cache might allow for user overrides on a plugin-specific level while still keeping the new directory priorities the same (in the original project, I'm pretty sure this is why the ability to prefix the function with the name of the dll was added: dgdecode_MPEG2Source vs. mpeg2dec3_MPEG2Source, even though that's the case of the same function, but different .dll names - having it user-customizable would remove the need for special prefixing since the user could set whichever plugin they want as the default, and it would cover cases if the plugin and function names were the same, since the user could have the ability to set an alias of their own in the cache). Or in the RemoveGrain example, it could use a separate CPU dispatching plugin or function and the cache could be used to prioritize which version would take priority on the host CPU.



Regarding FFMS2, I don't think the extra 2.6 stuff in its version of avisynth_c.h was preventing it from autoloading, since I experimented with it a little last night using Meld to merge the changes in locally, and it still didn't autoload. So it seems the problem lies elsewhere (although having avisynth_c.h get updated would be nice).
qyot27 is offline  
Old 3rd October 2013, 07:41   #76  |  Link
ultim
AVS+ Dev
 
ultim's Avatar
 
Join Date: Aug 2013
Posts: 359
Quote:
Originally Posted by qyot27 View Post
...
The cache idea was mostly thought up to avoid that scenario, since it would store the names and plugin locations of the functions and avoid having to perform those first two steps and any instability that could cause - instead, it'd read the cache, compare it to the script, and then load only the plugins used in the script.
...
Ah, now I understand. So you mean a separate cache instance per script. Yes, that should work, and makes sense as long as the cache format is chosen so that it is easily editable by the user. The problem I see though, is that the cache would need to be regenerated each time the contents of plugin folders are changed, including non-plugin files. Even if the script has not changed. This equals to rescanning all the plugins, we don't have to execute code from all of them though, only in new or changed files.

Quote:
Originally Posted by qyot27 View Post
Regarding FFMS2, I don't think the extra 2.6 stuff in its version of avisynth_c.h was preventing it from autoloading, since I experimented with it a little last night using Meld to merge the changes in locally, and it still didn't autoload. So it seems the problem lies elsewhere (although having avisynth_c.h get updated would be nice).
I didn't have time yesterday again, so all I could manage was to verify that the precompiled 2.17 Cplugin works (as a Cplugin, ofc). It works. Today I'm gonna recompile the cplugin from latest git and see what's the situation with that.
ultim is offline  
Old 3rd October 2013, 08:33   #77  |  Link
qyot27
...?
 
qyot27's Avatar
 
Join Date: Nov 2005
Location: Florida
Posts: 1,419
Quote:
Originally Posted by ultim View Post
I didn't have time yesterday again, so all I could manage was to verify that the precompiled 2.17 Cplugin works (as a Cplugin, ofc). It works. Today I'm gonna recompile the cplugin from latest git and see what's the situation with that.
Could it have to do with the dual handling of 2.5 and 2.6 (even though that's present in 2.17 also)? The C-plugin explicitly has a 2.5 mode and a 2.6 mode. You can see this in action by throwing FFmpeg into gdb and using a script using FFMS2 with either 2.5.8 or 2.6 - the language changes in the debug messages.

With the latest 2.6 MT (the video file being served in was 10-bit 4:4:4 H.264):
Code:
(gdb) r -i test.avs
...
  libpostproc    52.  3.100 / 52.  3.100
Error while mapping shared library sections:
`C:\WINDOWS\system32\avisynth.dll': not in executable format: File format not recognized
warning: FFMS2 avs plugin: Initializing...
warning: FFMS2 - avs 2.6 mode
Guessed Channel Layout for  Input Stream #0.1 : stereo
Input #0, avisynth, from 'test.avs':
  Duration: 00:05:00.47, start: 0.000000, bitrate: 0 kb/s
    Stream #0:0: Video: rawvideo (444P / 0x50343434), yuv444p, 848x480, 23.98 fps, 23.98 t
br, 23.98 tbn, 23.98 tbc
...
vs. 2.5.8:
Code:
(gdb) r -i test.avs
...
  libpostproc    52.  3.100 / 52.  3.100
warning: DllMain: hModule=0x10000000, ulReason=1, lpReserved=0x00000000, gRefCnt = 0

warning: FFMS2 avs plugin: Initializing...
warning: FFMS2 - avs 2.5 mode
Guessed Channel Layout for  Input Stream #0.1 : stereo
Input #0, avisynth, from 'test.avs':
  Duration: 00:05:00.47, start: 0.000000, bitrate: 0 kb/s
    Stream #0:0: Video: rawvideo (BGRA / 0x41524742), bgra, 848x480, 23.98 fps, 23.98 tbr,
 23.98 tbn, 23.98 tbc
...
I'm fairly sure I'm still using the same build I posted here, if it helps:
http://forum.doom9.org/showthread.php?p=1643738#post1643738

But building right from git would work, too. All the changes to the c_plugin branch after that build were for Linux/OSX (and the currently-disabled HEVC tag).

Last edited by qyot27; 3rd October 2013 at 08:54.
qyot27 is offline  
Old 3rd October 2013, 08:55   #78  |  Link
ultim
AVS+ Dev
 
ultim's Avatar
 
Join Date: Aug 2013
Posts: 359
Quote:
Originally Posted by qyot27 View Post
I can't quite remember *when* 2.17 was tagged, but I'm thinking it probably was from right before it gained the dual 2.5/2.6 handling. The C-plugin explicitly has a 2.5 mode and a 2.6 mode. You can see this in action by throwing FFmpeg into gdb and using a script using FFMS2 with either 2.5.8 or 2.6 - the language changes in the debug messages.
Yes I noticed that some builds have dual interfaces, in my case C and 2.5. I made sure that the C interface of ffms2 was used by temporarily changing the interface load order in Avs+, and ffms2 loaded up and worked correctly with the C interface. That was 2.17 though.

Quote:
Originally Posted by qyot27 View Post
I'm fairly sure I'm still using the same build I posted here, if it helps:
http://forum.doom9.org/showthread.php?p=1643738#post1643738

But building right from git would work, too. All the changes to the c_plugin branch after that build were for Linux/OSX (and the currently-disabled HEVC tag).
Oh, thx for the link. I'll just use that binary then instead of building it myself
ultim is offline  
Old 3rd October 2013, 09:04   #79  |  Link
qyot27
...?
 
qyot27's Avatar
 
Join Date: Nov 2005
Location: Florida
Posts: 1,419
Well, unfortunately, I think it's actually a problem with my computer. I went and tried the official 2.17-cplugin package, and it won't load either.

Code:
[avisynth @ 03792520] Script error: There is no function named 'ffindex'.
(C:/Program Files/AviSynth 2.5/plugins/FFMS2.avsi, line 32)
(test.avs, line 1)
test.avs: Unknown error occurred
or if I don't try to use FFMS2.avsi:
Code:
[avisynth @ 03792520] Script error: There is no function named 'FFVideoSource'.
(test.avs, line 1)
test.avs: Unknown error occurred
So something is going on that makes it fail, but I don't know what.
qyot27 is offline  
Old 3rd October 2013, 19:35   #80  |  Link
ultim
AVS+ Dev
 
ultim's Avatar
 
Join Date: Aug 2013
Posts: 359
I tried your build from http://forum.doom9.org/showthread.php?p=1643738#post1643738 , and it worked for me. If the problem on your computer is a missing dependency, depends.exe can help you diagnose what is wrong.
ultim is offline  
Closed Thread

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 15:46.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2024, vBulletin Solutions Inc.