Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Reply
 
Thread Tools Search this Thread Display Modes
Old 6th October 2013, 14:07   #101  |  Link
ultim
AVS+ Dev
 
ultim's Avatar
 
Join Date: Aug 2013
Posts: 359
Quote:
Originally Posted by vdcrim View Post
Just fixed in current git.

The $InternalFunctions$ and $PluginFunctions$ global vars are gone since the plugin management rewrite. Was that by mistake?
Ah cool, I just saw your post after posting my own. Would you be okay with modifications to AvsPmod? That means you might need a small path to recognize you are using Avisynth+, but I think it might even be posisble to find a unified way that works with both Avisynth and Avisynth+. I'll give some thougth to it if you are interested. Ofc I'll make the necessary modifications too.

Last edited by ultim; 6th October 2013 at 14:12.
ultim is offline   Reply With Quote
Old 6th October 2013, 14:40   #102  |  Link
ultim
AVS+ Dev
 
ultim's Avatar
 
Join Date: Aug 2013
Posts: 359
Regarding porting to x64:

Taking code from Avxsynth and Avisynth64 could indeed be very helpful, but one must be careful becuase these have been forked long ago and are not up to date with Avisynth 2.6. Meaning the funcationality and semnatics of the filters might have diverged.

x64 VC++ does not support inline asm at all, which means handoptimized pieces of code will either need to be compiled separately using an external assembler, or they need to be converted to intrinsics. IMHO, intrinsics are way better. First of all many/most intel-style intrinsics are portable between compilers (even between GCC, and MSVC), and second, having to compile external .asm files using a dedicated assembler would complicate the build process a lot. Not to mention CMake ATM doesn't even have support for that when generating VS project files. So, TurboPascal7, if you con really help porting to intrinsics, that would be more than great.

Porting to x64 could even be made step-by-step. Most ASM resides in builtin filters, so as long as the core without filters can be compiled in x64, we can be adding more and more builtin filters back to an already working x64 version as the porting continues.

As a separate note, I see a potential problem of maintaining a unified public avisynth.h header without breaking 32-bit plugins. We could ofc always maintain two totally separate versions of that header, but if possible we should try to avoid that.

Last edited by ultim; 6th October 2013 at 14:43.
ultim is offline   Reply With Quote
Old 6th October 2013, 14:56   #103  |  Link
TurboPascal7
Registered User
 
TurboPascal7's Avatar
 
Join Date: Jan 2010
Posts: 270
There are some problems with intrinsics though:
1) I'm not sure how well they are supported in vs2005 and 2008. While 2008 is probably okay, 2005 might have some serious issues. I haven't worked with it for years and didn't do any C++ back then so I'm not sure.
2) VC++ doesn't support MMX intrinsics in x64 mode. So one either #ifdefs every MMX routine or drops MMX optimization altogether (I do approve dropping it but I'm not sure how many avisynth users don't even have a P4 these days).
TurboPascal7 is offline   Reply With Quote
Old 6th October 2013, 15:24   #104  |  Link
jpsdr
Registered User
 
Join Date: Oct 2002
Location: France
Posts: 1,764
If ASM reside most in built-in filter and not in the core, maybe what's allready done with avisynth64 could be used, assuming that differences with actual version are more in the core than filters...
jpsdr is offline   Reply With Quote
Old 6th October 2013, 15:24   #105  |  Link
ultim
AVS+ Dev
 
ultim's Avatar
 
Join Date: Aug 2013
Posts: 359
Quote:
Originally Posted by TurboPascal7 View Post
1) I'm not sure how well they are supported in vs2005 and 2008. While 2008 is probably okay, 2005 might have some serious issues. I haven't worked with it for years and didn't do any C++ back then so I'm not sure.
Well, I was supporting VS2005 because there was no need to drop it. If it does not have proper support for the intrinsics we need (thereby stepping in the way of 64bit support), that'd be enough reason for me to drop it. But by looking at this page, VC++ 2005 does seem to have intrinsics support for SSE2. And if not, I'm okay to drop it.

Quote:
Originally Posted by TurboPascal7 View Post
2) VC++ doesn't support MMX intrinsics in x64 mode. So one either #ifdefs every MMX routine or drops MMX optimization altogether (I do approve dropping it but I'm not sure how many avisynth users don't even have a P4 these days).
We could #ifdef around those parts for 32-bit users on old machines. In the x64-builds, that wouldn't matter anyway coz the presence of SSE2 is guaranteed.

I think I need to add a dynamic dispatch ability to Avisynth+ that filters and plugins can make use of
ultim is offline   Reply With Quote
Old 6th October 2013, 15:31   #106  |  Link
TurboPascal7
Registered User
 
TurboPascal7's Avatar
 
Join Date: Jan 2010
Posts: 270
One more important question you are aware of is alignment. Did you decide to keep it the way avisynth works now or provide any reasonable guarantee?

I'm not sure that you need to add anything on the core level to help dispatching. Providing a method to get CPU flags (the way it is now) is enough to make filters implementation as simple as possible. I'm not sure how you could simplify it even more.
TurboPascal7 is offline   Reply With Quote
Old 6th October 2013, 16:07   #107  |  Link
aegisofrime
Registered User
 
Join Date: Apr 2009
Posts: 462
I'm not a developer so I can't offer any help, just a normal user, but I just want to say thank you to ultim for initiating this project, as well as other contributors. This is probably the most exciting thing on the Avisynth in my few years of video processing.
aegisofrime is offline   Reply With Quote
Old 6th October 2013, 16:56   #108  |  Link
qyot27
...?
 
qyot27's Avatar
 
Join Date: Nov 2005
Location: Florida
Posts: 1,084
Quote:
Originally Posted by ultim View Post
We could #ifdef around those parts for 32-bit users on old machines. In the x64-builds, that wouldn't matter anyway coz the presence of SSE2 is guaranteed.
I'm definitely in favor of ifdeffing around it. My main setup is a Pentium III-era Celeron, and this was a point of contention that arose in VapourSynth. The stated solution: use AviSynth if you've got an ancient CPU.

The alternative would be to just disable asm completely on non-SSE2 CPUs, but if there's benefits to be had from MMX and SSE I wouldn't throw it out completely.

Last edited by qyot27; 6th October 2013 at 17:00.
qyot27 is offline   Reply With Quote
Old 6th October 2013, 17:43   #109  |  Link
ultim
AVS+ Dev
 
ultim's Avatar
 
Join Date: Aug 2013
Posts: 359
Quote:
Originally Posted by qyot27 View Post
I'm definitely in favor of ifdeffing around it. My main setup is a Pentium III-era Celeron, and this was a point of contention that arose in VapourSynth. The stated solution: use AviSynth if you've got an ancient CPU.

The alternative would be to just disable asm completely on non-SSE2 CPUs, but if there's benefits to be had from MMX and SSE I wouldn't throw it out completely.
I do agree with Myrsloik that programming for Pentium3 is not worth it nowadays, but I see no harm in keeping existing code there for it. All we need to do is put a define around it so that the compiler ignores it in 64-bit builds, while it can still build in 32-bit versions. This works at least for "stable" filters. Others could have a problem, that when they are updated, the filter maintainer would have to update the MMX routines too, and there probably won't be too much motivation for that.
ultim is offline   Reply With Quote
Old 6th October 2013, 17:48   #110  |  Link
ultim
AVS+ Dev
 
ultim's Avatar
 
Join Date: Aug 2013
Posts: 359
Quote:
Originally Posted by TurboPascal7 View Post
One more important question you are aware of is alignment. Did you decide to keep it the way avisynth works now or provide any reasonable guarantee?
Some changes that I proposed for inclusion in my very first changeset include a refactor of alignment-related functionality. Let me try to make a very-very cautious statement, that I think a minimum alignment guarantee of 32 bytes should be there, unless a script uses unaligned crop.

Getting rid of unaligned crops would undoubtably strengthen this semi-guarantee, but I also must add that there might be other places I have overlooked. Should you find any violation of this alignment, I'll get rid of it for sure as soon as I get to know of it.

Last edited by ultim; 6th October 2013 at 17:51.
ultim is offline   Reply With Quote
Old 6th October 2013, 18:19   #111  |  Link
ARDA
Registered User
 
Join Date: Nov 2001
Posts: 291
Quote:
Originally Posted by ultim
x64 support is also a goal, but certianly not for the near-future. As many of you have said,
the greatest obstacle here are the pieces of inline asm. I must admit that I'm not very versed
in x86 asm, so here too I will be counting on your help.
Quote:
Originally Posted by ultim
x64 VC++ does not support inline asm at all, which means handoptimized pieces of code will either
need to be compiled separately using an external assembler, or they need to be converted to intrinsics.
IMHO, intrinsics are way better. First of all many/most intel-style intrinsics are portable between compilers
(even between GCC, and MSVC), and second, having to compile external .asm files using a dedicated assembler
would complicate the build process a lot. Not to mention CMake ATM doesn't even have support for that when
generating VS project files. So, TurboPascal7, if you con really help porting to intrinsics,
that would be more than great.
Quote:
Originally Posted by ultim
Porting to x64 could even be made step-by-step. Most ASM resides in builtin filters, so as long as the core
without filters can be compiled in x64, we can be adding more and more builtin filters back to an already
working x64 version as the porting continues.
Quote:
Originally Posted by TurboPascal7
VC++ doesn't support MMX intrinsics in x64 mode
Some fast thoughts not ordered by priorities;

To compile assembly code with an external assembler is not so hard, at least according to my experience
Nasm/Yasm syntax is very near inline asm, except some pointers syntax, not too much.
They are also portable between enviroments and platforms if I am not wrong. Well, I work almost always
with Microsoft Visual Studio C++ 2008 and Yasm or Nasm.

The idea of separate many filters out of core to port them to x64 is a good one, and for 32
bit version as well. I always dreamt with a core very light, the lightest possible, and different
projects of filters grouped by their functions, tweak colors, editing, resizing, etc.
It could help to bring more developers with some specific knowledge that don't dare to manage the whole
project, besides that it could make easier to track bugs in the core or in a specifig group of filters.
Many years ago I posted something similar, but it seems there is a limit number of dll that can be loaded.
http://forum.doom9.org/showthread.ph...052#post698052

In short if possible I wouldn't include them back to core.

I also think there is a lot work previous port to x64, and I agree is not for near future.
And in that future probably mmx or Isse will not be needed anymore, while 32 bits it doesn't hurt
to mantain mmx and mainly Isse cause the profit of non temporal store instructions.
ARDA is offline   Reply With Quote
Old 6th October 2013, 18:20   #112  |  Link
qyot27
...?
 
qyot27's Avatar
 
Join Date: Nov 2005
Location: Florida
Posts: 1,084
I don't think it's worth focusing on computers as old as mine either. It's more that I really couldn't see why a simple --disable-asm option wouldn't have sufficed if running the asm on a non-SSE2 CPU would cause problems. I just couldn't grasp how whether the core could use X certain SIMD set would matter to a plugin author, because how I understand the way this all works, the plugin would simply use the desired core function at whatever level of optimization was there. If it was boosted by SSE2 or whatever, great, if not, then it would fall back to the plain C or C++ version and if the user's on a computer old enough for that to be true, then they already know how slow their computer is, so I still don't see the problem in merely letting them run it in an unoptimized state.
qyot27 is offline   Reply With Quote
Old 6th October 2013, 18:25   #113  |  Link
Groucho2004
Cantankerous Fossil
 
Groucho2004's Avatar
 
Join Date: Mar 2006
Location: A wretched hive of scum and villainy
Posts: 4,470
Quote:
Originally Posted by ARDA View Post
I always dreamt with a core very light, the lightest possible, and different projects of filters grouped by their functions, tweak colors, editing, resizing, etc.
This would be the ideal scenario in my book as well but I don't know how much work would be involved to dig out the core and then add these - let's call them native - filters.
Groucho2004 is offline   Reply With Quote
Old 6th October 2013, 20:25   #114  |  Link
TheFluff
Excessively jovial fellow
 
Join Date: Jun 2004
Location: rude
Posts: 1,073
Quote:
Originally Posted by ultim View Post
Well, I was supporting VS2005 because there was no need to drop it. If it does not have proper support for the intrinsics we need (thereby stepping in the way of 64bit support), that'd be enough reason for me to drop it. But by looking at this page, VC++ 2005 does seem to have intrinsics support for SSE2. And if not, I'm okay to drop it.
VC++2005 supports it but the code generation apparently has some minor issues, and MMX is worse off than SSE. Avery Lee has written about intrinsics in VC++ at some length, see for example:
http://virtualdub.org/blog/pivot/entry.php?id=46
http://virtualdub.org/blog/pivot/entry.php?id=301
http://virtualdub.org/blog/pivot/entry.php?id=340
TheFluff is offline   Reply With Quote
Old 6th October 2013, 23:49   #115  |  Link
TurboPascal7
Registered User
 
TurboPascal7's Avatar
 
Join Date: Jan 2010
Posts: 270
Quote:
Originally Posted by qyot27 View Post
If it was boosted by SSE2 or whatever, great, if not, then it would fall back to the plain C or C++ version and if the user's on a computer old enough for that to be true, then they already know how slow their computer is, so I still don't see the problem in merely letting them run it in an unoptimized state.
But this requires filters/plugins to have this unoptimized plain C version, which might not always be the case (take a look a sangnom2 for example, which doesn't have a plain C version. Original sangnom didn't either). I dunno about you, but I don't really see any point in adding that only to support some 15-years old CPUs (readability problem aside). This is probably not the problem for internal filters though, since most (all?) of them do have the C path.

Guarantees is a good thing and IMHO requiring a newer-than-10-years-old CPU if this makes plugin authors' lifes easier is reasonable.

Quote:
Originally Posted by Groucho2004 View Post
This would be the ideal scenario in my book as well but I don't know how much work would be involved to dig out the core and then add these - let's call them native - filters.
The core filters are separated quite well and moving them to an external dll wouldn't be hard.

That said, I can't say I fully support the idea. I believe the core itself must provide basic video operations the way it does now and I don't see how separating them would help anything. Especially when compilation of the core is dead simple and fast now.

Quote:
Originally Posted by ultim View Post
Some changes that I proposed for inclusion in my very first changeset include a refactor of alignment-related functionality. Let me try to make a very-very cautious statement, that I think a minimum alignment guarantee of 32 bytes should be there, unless a script uses unaligned crop.

Getting rid of unaligned crops would undoubtably strengthen this semi-guarantee, but I also must add that there might be other places I have overlooked. Should you find any violation of this alignment, I'll get rid of it for sure as soon as I get to know of it.
Here we come to the problem I originally tried to explain to IanB (and failed) - only strong guarantees are good for developers.
Take the core filters for example. Right now, most of them are optimized for MMX only, which does not require any type of alignment the way SSE2 do. While porting to x64 and SSE2 (which is the only option if we use intrinsics), one will need to take alignment into account. And without any strong guarantee, he would essentially need to either
1) Template almost every SSE2-optimized routine ever to use unaligned or aligned loads based on provided data. This becomes even worse if we use external asm because simple templates are much easier than any kind of asm macros.
2) Always use unaligned loads which makes every single SSE2-optimized filter slower only to provide faster crop and possibly faster source filters which I'm not aware of.

That said, I agree we shouldn't make rush decisions. This is indeed a hard question that might (and probably will) break some older filters. I guess I'll do a build with alignment parameter to NewVideoFrame ignored and see how much breaks later.

----
Also, setting up an IRC channel on freenode/rizon or something to speed up the communication process would be nice, I guess (there's already #avisynth on freenode btw).

Last edited by TurboPascal7; 7th October 2013 at 02:56.
TurboPascal7 is offline   Reply With Quote
Old 7th October 2013, 07:52   #116  |  Link
ultim
AVS+ Dev
 
ultim's Avatar
 
Join Date: Aug 2013
Posts: 359
Separating *all* the filters out of core is not my plan either, actually. I already made external plugins out of many functions that IMHO do not belong into the core, like DirectShowSource, audio timestrething, support for vdub filters etc, and maybe I'll do the same with a few more, but I think too that the core library should be usable on its own too. When I spoke about a step-by-step porting process with a growing set of x64 filters, I was thinking of #ifdef-ing around filter code and gradually enabling them in builds as porting advances.

I *do* support moving seldomly used filters out of the core, so reviewing what belongs there and what not could go onto the TODO list, but I too don't think that the core sould be totally bare and empty of filters. Moving them out, as TurboPascal7 mentioned, is not hard. It is actually so easy that it could be an introductory exercise for anybody wanting to contribute.

ARDA mentioned a DLL loading limit in Avisynth: in the most recent version of the non-plus Avisynth, it is 50. This limit has already been removed from Avisynth+. It was artificial and not-needed in the first place, and it probably got introduced because people were compiling plugins with a static CRT, in which case Windows imposes a limit of 128 loaded DLLs (at least that's the observable effect, the true reason lies elsewhere, but that's a different story). Anyway, my point is, as long as our plugins are linked against a dynamic CRT, having many DLLs should not pose a problem.
ultim is offline   Reply With Quote
Old 7th October 2013, 08:02   #117  |  Link
ultim
AVS+ Dev
 
ultim's Avatar
 
Join Date: Aug 2013
Posts: 359
Quote:
Originally Posted by TurboPascal7 View Post
Here we come to the problem I originally tried to explain to IanB (and failed) - only strong guarantees are good for developers
I understand and agree with you fully. The reason I didn't give a strong guarantee is not because I don't want to introduce one, it is because I'm not sure that it is currently fully fulfilled. I also want a proper alignment guarantee to be there, which is why I'm going to correct any left-over pieces of code that violate this, if any is still found. I think avoiding unaligned crops brings pretty far in Avisynth+, and definetely further than plain Avisynth.

Quote:
Originally Posted by TurboPascal7 View Post
Also, setting up an IRC channel on freenode/rizon or something to speed up the communication process would be nice, I guess (there's already #avisynth on freenode btw).
I gues we could. Actually I am regularly on rizon anyway (and when not, you'll find my bouncer), but because there are more people accustomed to the forums than to IRC, I find a forum a better place for discussions. If we used IRC, many wouldn't be able to follow.
ultim is offline   Reply With Quote
Old 7th October 2013, 08:28   #118  |  Link
ultim
AVS+ Dev
 
ultim's Avatar
 
Join Date: Aug 2013
Posts: 359
On the intrinsics vs. asm topic, I will only say that intrinsics are more favorable to me personally, but I'll certainly accept contributed code written in asm too. I won't reject the integration of asm files just because they are not using intrinsics, as long as the contributor also provides a working integration into the current build system. So I say if the contributor is indifferent about using intrinsics or asm, please use intrinsics, but your contribution will be greatly appreciated otherwise too.

Intrinsics integrate more naturally into C compilers and into the existing build system. But maybe most importantly, pure intrinsics can be uniformly compiled into both 32bit and 64bit code, so they win by large in maintainability too. Many prefer asm because they find it easier to read, that is also important, but it is subjective, so I'll leave the choice up to you.

True, intrinsics do not always generate optimal code, but it will be close, and it will still be much better than traditional C code. And let's be honest, hand-written ASM is not guaranteed to be optimal either. Not to mention, code generation from intrinsics improves with better compilers.

But anyway, if you don't want to use intrinsics, that's fine by me too.

Last edited by ultim; 7th October 2013 at 08:31.
ultim is offline   Reply With Quote
Old 7th October 2013, 10:48   #119  |  Link
ultim
AVS+ Dev
 
ultim's Avatar
 
Join Date: Aug 2013
Posts: 359
What kind of for-construct would you like to see?
Gavino's "for" looks like
Quote:
for ( variable = init , limit , step ) {
statements
}
where limit and step must evaluate to integers and are only evaluated once. There is also some built-in trickery whether the loop is up- or down-counting (see original thread for reference).

The alternative would be a C-style loop, common not only to C but most other computer languages.
Quote:
for ( statement1 , condition , statement2 ) {
statements
}
which strictly speaking is just a short form for
Quote:
statement1
while ( condition ) {
statements
statement2
}
I can see upsides and downsides too to Gavino's version. It might be easier to grasp for non-programmers, and also it gives Avisynth some possibilities to recognize infinite-loops, and helps prevent the user from shooting himself in the foot. C-style for-loops could then be emulated using manually written while-loops. On the other hand, it is less generic, non-standard, and despite all the simplifications and built-in automatic, it still cannot prevent all user errors, like when the loop variable is modified inside the body. Programmers could also argue that it is counter-intuitive, because 'limit' and 'step' are only evaluated once even if their value is dynamic.

So, my question is simple: Do you want Gavino-style for-loops or C-style for-loops in Avisynth+?
ultim is offline   Reply With Quote
Old 7th October 2013, 11:00   #120  |  Link
mirkosp
Registered User
 
Join Date: Jul 2009
Posts: 18
It's not like they are needed, since they can already be done through recursion if one needs to.
Let's not forget that AviSynth does NOT exist as a normal programming language, but more as a NLE tool; I doubt this nature should change with a fork. As such, common programming structures don't really need to be defined.
If one really needs to employ many loops for a script, it's likely because they need to write something complex, and it would be better to straight out write that in a proper language and build a .dll for use.
mirkosp is offline   Reply With Quote
Reply

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 19:26.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2019, vBulletin Solutions Inc.