Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Development

Reply
 
Thread Tools Search this Thread Display Modes
Old 23rd July 2016, 18:20   #21  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: Los Angeles, California
Posts: 2,119
Quote:
One thing I'm thinking is that if one would want to use some sort of advanced neural network algorithm, or some sort of AI, for highly advanced processing, these sorts of processing could be more easily done in a higher-level language like C# and linking to a library to do the neural processing (or whatever that would be). (That example is hypothetical)
???? no one does algorithm related stuff in complicated languages like c#, and of course not something like c++ either
Algorithms are developed in very high level scripting dynamic languages like Matlab and Python, and then ported to c/c++ for performance reasons
__________________
If I got new ideas, will post here: https://github.com/IFeelBloated
feisty2 is offline   Reply With Quote
Old 24th July 2016, 02:36   #22  |  Link
MysteryX
Soul Architect
 
MysteryX's Avatar
 
Join Date: Apr 2014
Posts: 2,040
Quote:
Originally Posted by feisty2 View Post
???? no one does algorithm related stuff in complicated languages like c#, and of course not something like c++ either
Algorithms are developed in very high level scripting dynamic languages like Matlab and Python, and then ported to c/c++ for performance reasons
I didn't know Python had a strength for mathematical algorithms.

There is no such thing as "nobody". If I was to design a complicated algorithm, I would first do it in a high-level language like... pen and paper. Then transcribe it into C# or C++ depending on the needs.
MysteryX is offline   Reply With Quote
Old 24th July 2016, 02:50   #23  |  Link
MysteryX
Soul Architect
 
MysteryX's Avatar
 
Join Date: Apr 2014
Posts: 2,040
A quick search in Google showed the pros/cons of Python over C#. Nowhere it mentions it being a better language for mathematical algorithms; it's just a simpler language, kind of like Visual Basic used to be. That simplicity also comes at a small performance cost compared to C#. It's often more compared to JavaScript.

The learning curve for C# is quite steep but once you know it it's a simple language to use.

Quote:
Python is winner in: ease of learning, cross platform development, availability of open source libraries
C# is winner in: standard library, language features, development process and tools, performance, language evolution speed
Roughly even: syntax (Python is better in readability, C# has more consistent syntax), adoption.
I'd be curious to know how much optimizations .NET can do for algorithms running on raw memory streams. It can make use of all the CPU-specific commands because it does the final compilation on-the-fly, but how well does it do it? How would it compare to C++, or to SSE2 assembly? This would be an advanced question for another forum.

To be clear: this thread is not about which language is better than any other, but about how to make the best use of C#. As to which language is best, my answer is simple: the one you're familiar with.

Last edited by MysteryX; 24th July 2016 at 03:05.
MysteryX is offline   Reply With Quote
Old 24th July 2016, 04:23   #24  |  Link
MysteryX
Soul Architect
 
MysteryX's Avatar
 
Join Date: Apr 2014
Posts: 2,040
It's really not as bad as I thought regarding C# for raw math processing.

First, C# does support raw pointers when compiling with /unsafe.

Second, SIMD-optimized vector types have been added in .NET 4.6

Someone may have to take some time to develop a SIMD-optimized sample using raw pointers and comparing the performance with C++.
MysteryX is offline   Reply With Quote
Old 24th July 2016, 04:44   #25  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: Los Angeles, California
Posts: 2,119
Quote:
Originally Posted by MysteryX View Post
I didn't know Python had a strength for mathematical algorithms.
https://en.wikipedia.org/wiki/SciPy
https://en.wikipedia.org/wiki/NumPy
https://github.com/gwpy/gwpy
__________________
If I got new ideas, will post here: https://github.com/IFeelBloated
feisty2 is offline   Reply With Quote
Old 24th July 2016, 04:50   #26  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: Los Angeles, California
Posts: 2,119
Quote:
Originally Posted by MysteryX View Post
To be clear: this thread is not about which language is better than any other, but about how to make the best use of C#. As to which language is best, my answer is simple: the one you're familiar with.
not saying py is a better language than c#, simply pointing out c# is not widely used for scientific purposes (including algorithm designing) and py is
__________________
If I got new ideas, will post here: https://github.com/IFeelBloated
feisty2 is offline   Reply With Quote
Old 24th July 2016, 05:10   #27  |  Link
MysteryX
Soul Architect
 
MysteryX's Avatar
 
Join Date: Apr 2014
Posts: 2,040
The scientific community is a whole other community than the programming community, so it's possible that they favor Python; but I'm not part of the scientific community.

SciPy and NumPy have however been ported to .NET and are options to be explored. Does this have SIMD optimization? How does this compare to SIMD-optimized .NET types?
https://www.infoq.com/news/2011/07/NumPy-NET
http://pytools.codeplex.com/wikipage...y%20for%20.Net

Last edited by MysteryX; 24th July 2016 at 05:12.
MysteryX is offline   Reply With Quote
Old 24th July 2016, 05:25   #28  |  Link
feisty2
I'm Siri
 
feisty2's Avatar
 
Join Date: Oct 2012
Location: Los Angeles, California
Posts: 2,119
then you're trolling, first you mentioned about "advanced neural network algorithm, or some sort of AI, for highly advanced processing", which are obviously scientific (algorithm developing) topics, then you said "You're not part of the scientific community" so you don't really give a, so what's the point?
people write prototype of the algorithm in scripting languages, and then port the prototype to a lower level language (c or c++ or stuff like that) and get a performance boost, they simply don't merge the "developing" stage and "porting" stage as one cuz that's gonna be a mess
__________________
If I got new ideas, will post here: https://github.com/IFeelBloated
feisty2 is offline   Reply With Quote
Old 24th July 2016, 06:00   #29  |  Link
MysteryX
Soul Architect
 
MysteryX's Avatar
 
Join Date: Apr 2014
Posts: 2,040
It's not more difficult to program it in C# than in Python if you're using the same core processing library.

The difference is that Python is easier to get started with. Thus, for people whose primary job is not programming, it is easier to use. For someone who already knows C#, then there's no point in first doing it in Python.

C# is a lot simpler and clearer than C++.

Also, if C# can provide good SIMD performance using either Python Tools or SIMD data types, then it would make it a whole lot easier to optimize code properly. C++ intrinsics have a few downside:
- Complicated to use and require deep knowledge of assembly
- The same code must be written several times: CPU, SSE2, AVX, etc.
- Very difficult to maintain and/or debug
- If AVX SIMD is used, the DLL must be compiled for AVX and will fail on other CPUs; thus several DLL versions have to be produced. Very unpractical for deployment.
- Most importantly, I can't write it myself

If one of the C# options can provide performance that isn't too far, but with a *much* simpler syntax, it would allow for SIMD optimization in many places where it wouldn't be done otherwise.

Last edited by MysteryX; 24th July 2016 at 06:30.
MysteryX is offline   Reply With Quote
Old 25th July 2016, 18:00   #30  |  Link
MysteryX
Soul Architect
 
MysteryX's Avatar
 
Join Date: Apr 2014
Posts: 2,040
Nowhere can I read that SciPy and NumPy have SIMD optimization, so SIMD-enabled vector types are the way to go.
MysteryX is offline   Reply With Quote
Old 18th August 2016, 14:33   #31  |  Link
MysteryX
Soul Architect
 
MysteryX's Avatar
 
Join Date: Apr 2014
Posts: 2,040
Here's something else interesting I found. It is possible to embed a C++ DLL within a C# DLL. This would allow the simplicity of C# while also being able to use C++ assembly without requiring a separate DLL. The best of both worlds!

http://stackoverflow.com/questions/6...ed-c-sharp-dll
https://github.com/Fody/Costura

So in practice, one can do it in C#, and if later he wants to add assembly optimization, it can also be done.
MysteryX is offline   Reply With Quote
Old 19th August 2016, 09:50   #32  |  Link
junh1024
Registered User
 
Join Date: Mar 2011
Posts: 16
Quote:
Originally Posted by MysteryX View Post
Here's something else interesting I found. It is possible to embed a C++ DLL within a C# DLL. This would allow the simplicity of C# while also being able to use C++ assembly without requiring a separate DLL. The best of both worlds!

http://stackoverflow.com/questions/6...ed-c-sharp-dll
https://github.com/Fody/Costura

So in practice, one can do it in C#, and if later he wants to add assembly optimization, it can also be done.
It's not really 'write in C# then optimize it later' like cython or w/e, it's

CLR needs to extract the embedded native DLL somewhere (Windows needs to have a file for the DLL to load it - it cannot load an image from raw memory)

(via http://stackoverflow.com/questions/6...ed-c-sharp-dll )

So even if you want to do DLLception and save 1 DLL, you'll need to extract the C++ part when you use it, leaving you with 2 DLLs, which is no saving at all.
junh1024 is offline   Reply With Quote
Old 19th August 2016, 10:38   #33  |  Link
TheRyuu
warpsharpened
 
Join Date: Feb 2007
Posts: 788
Quote:
Originally Posted by MysteryX View Post
- Very difficult to maintain and/or debug
If it's well written I don't think it should be any more difficult to maintain or debug provided equal knowledge in the languages used.

Quote:
Originally Posted by MysteryX View Post
- If AVX SIMD is used, the DLL must be compiled for AVX and will fail on other CPUs; thus several DLL versions have to be produced. Very unpractical for deployment.
If you're writing custom assembly/intrinsics you have the ability to do dispatching or use function pointers. It's not difficult to do runtime CPU detection.
TheRyuu is offline   Reply With Quote
Old 19th August 2016, 12:18   #34  |  Link
MysteryX
Soul Architect
 
MysteryX's Avatar
 
Join Date: Apr 2014
Posts: 2,040
Quote:
Originally Posted by junh1024 View Post
It's not really 'write in C# then optimize it later' like cython or w/e, it's

CLR needs to extract the embedded native DLL somewhere (Windows needs to have a file for the DLL to load it - it cannot load an image from raw memory)

(via http://stackoverflow.com/questions/6...ed-c-sharp-dll )

So even if you want to do DLLception and save 1 DLL, you'll need to extract the C++ part when you use it, leaving you with 2 DLLs, which is no saving at all.
The 2nd link I gave automates that process so it's done in the background. It needs to be configured right once, and then can be copy/pasted for any other project.
https://github.com/Fody/Costura

Quote:
Originally Posted by TheRyuu View Post
If you're writing custom assembly/intrinsics you have the ability to do dispatching or use function pointers. It's not difficult to do runtime CPU detection.
Chikuzen did that for some of my code. CPU detection for SSE2, and for AVX, it still required a separate DLL compilation.
MysteryX is offline   Reply With Quote
Old 19th August 2016, 12:55   #35  |  Link
Chikuzen
typo lover
 
Chikuzen's Avatar
 
Join Date: May 2009
Posts: 597
Quote:
Originally Posted by MysteryX View Post
Chikuzen did that for some of my code. CPU detection for SSE2, and for AVX, it still required a separate DLL compilation.
I noticed the way to avoid that in the evening, yesterday.
I'll send a patch in weekend.
__________________
my repositories
Chikuzen is offline   Reply With Quote
Old 19th August 2016, 16:08   #36  |  Link
MysteryX
Soul Architect
 
MysteryX's Avatar
 
Join Date: Apr 2014
Posts: 2,040
Quote:
Originally Posted by Chikuzen View Post
I noticed the way to avoid that in the evening, yesterday.
I'll send a patch in weekend.
Can you elaborate?
MysteryX is offline   Reply With Quote
Old 19th August 2016, 18:22   #37  |  Link
Chikuzen
typo lover
 
Chikuzen's Avatar
 
Join Date: May 2009
Posts: 597
Quote:
Originally Posted by MysteryX View Post
Can you elaborate?
it's possible if you can read Japanese.
__________________
my repositories
Chikuzen is offline   Reply With Quote
Old 20th August 2016, 00:19   #38  |  Link
TheRyuu
warpsharpened
 
Join Date: Feb 2007
Posts: 788
Quote:
Originally Posted by MysteryX View Post
Chikuzen did that for some of my code. CPU detection for SSE2, and for AVX, it still required a separate DLL compilation.
You can separate out the AVX functions into a separate file. You can then use file specific compilation flags which will only enable AVX for that file.
TheRyuu is offline   Reply With Quote
Old 1st September 2016, 17:03   #39  |  Link
MysteryX
Soul Architect
 
MysteryX's Avatar
 
Join Date: Apr 2014
Posts: 2,040
http://stackoverflow.com/questions/2...can-while-mono

SIMD support is from RyuJIT which currently only works for x64. For Avisynth we clearly need code that works on both x86 and x64.

Utility functions are very easy to write with C#, but for functions doing CPU-intensive operations on the frames, C++ is still best; unless C# allows easy SIMD integration.

Quote:
RyuJIT is based off of the same codebase as the x86 JIT. While it’s only for x64 right now, it’s a modern compiler that will be the basis of all our JIT compilers in the future: x86, ARM, MDIL and whatever else comes along. Having a single codebase means that .NET programs are more consistent between architectures—put another way, you generally get bug-for-bug compatibility. But having a single codebase also means we can innovate faster and bring you more code generation features more quickly.
This quote suggests that RyuJIT will eventually be used for all platforms once it is stable enough.

Until then, the best option is C++ or C# with an embedded C++ library to do the raw frame processing operations.
MysteryX is offline   Reply With Quote
Old 31st December 2016, 13:01   #40  |  Link
amayra
Quality Checker
 
amayra's Avatar
 
Join Date: Aug 2013
Posts: 195
Quote:
Originally Posted by amayra View Post
thanks AFN 1.0.2 work like a charm i have no issues until now
well i was wrong as usual
can you give examples for any filter please
__________________
I love Doom9
amayra is offline   Reply With Quote
Reply

Tags
.net, avisynth, filter

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 05:10.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2018, vBulletin Solutions Inc.