Welcome to Doom9's Forum, THE in-place to be for everyone interested in DVD conversion.

Before you start posting please read the forum rules. By posting to this forum you agree to abide by the rules.

 

Go Back   Doom9's Forum > Capturing and Editing Video > Avisynth Usage

Reply
 
Thread Tools Search this Thread Display Modes
Old 29th June 2016, 15:10   #21  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
Quote:
Originally Posted by MysteryX View Post
Didn't he say he's buffering the video to facilitate playback and that caused memory to run out?
Oh, that. I was thrown off a bit by your choice of terminology.
__________________
Groucho's Avisynth Stuff
Groucho2004 is offline   Reply With Quote
Old 29th June 2016, 15:57   #22  |  Link
MysteryX
Soul Architect
 
MysteryX's Avatar
 
Join Date: Apr 2014
Posts: 2,552
Quote:
Originally Posted by Groucho2004 View Post
Oh, that. I was thrown off a bit by your choice of terminology.
Don't be thrown off so easily. No I wasn't talking about AviSynth's internal cache.

kagetoki, .NET and Software Architecture are my specialties. If you have any questions about how to structure the code, just ask.

Running the script in a different thread does have great advantages
1. Main app can cache without running out of memory
2. If AVS process crashes or freezes, it won't affect the main app and the main app can kill it
3. You can run both x64 and x86 scripts. Otherwise, a x86 app can only run x86 scripts and a x64 app can only run x64 scripts.

It really shouldn't be hard to implement at all. With C++, this would be a heck of a nightmare to setup cross-process communication. With C#, you have many options available to do it in a clean and simple way. C# also makes multi-threading and asynchronous operations very easy to code, make good use of it.

As for the database, I see you store it in a XML file. You might want to consider using SQLite instead. Then you can manipulate your data with LINQ and it's easy to expand your data structure as the needs evolve.

You might want to compile the application with Target Platform: Any CPU (project properties, Build). That way, it will run in x86 on x86 systems and in x64 on x64 systems. Managed code will work fine and the restriction then is that you cannot use x86-specific components. The only x86 components you should use are to run AVS and that can be in the other process. The script can run in x86, pass the computed frames through Named Pipes to the main app running in x64 that can cache those without worrying about memory.

By the way, is this an open source or closed source project?

Last edited by MysteryX; 29th June 2016 at 16:22.
MysteryX is offline   Reply With Quote
Old 29th June 2016, 16:37   #23  |  Link
StainlessS
HeartlessS Usurer
 
StainlessS's Avatar
 
Join Date: Dec 2009
Location: Over the rainbow
Posts: 10,227
Quote:
Originally Posted by MysteryX View Post
By the way, is this an open source or closed source project?
Full C# source via Sourceforge link in 1st post, or here:- https://sourceforge.net/projects/avs...ble/01024/src/
__________________
I sometimes post sober.
StainlessS@MediaFire ::: AND/OR ::: StainlessS@SendSpace

"Some infinities are bigger than other infinities", but how many of them are infinitely bigger ???
StainlessS is offline   Reply With Quote
Old 29th June 2016, 17:15   #24  |  Link
real.finder
Registered User
 
Join Date: Jan 2012
Location: Mesopotamia
Posts: 2,532
Quote:
Originally Posted by MysteryX View Post
Don't be thrown off so easily. No I wasn't talking about AviSynth's internal cache.

kagetoki, .NET and Software Architecture are my specialties. If you have any questions about how to structure the code, just ask.

Running the script in a different thread does have great advantages
1. Main app can cache without running out of memory
2. If AVS process crashes or freezes, it won't affect the main app and the main app can kill it
3. You can run both x64 and x86 scripts. Otherwise, a x86 app can only run x86 scripts and a x64 app can only run x64 scripts.

It really shouldn't be hard to implement at all. With C++, this would be a heck of a nightmare to setup cross-process communication. With C#, you have many options available to do it in a clean and simple way. C# also makes multi-threading and asynchronous operations very easy to code, make good use of it.

As for the database, I see you store it in a XML file. You might want to consider using SQLite instead. Then you can manipulate your data with LINQ and it's easy to expand your data structure as the needs evolve.

You might want to compile the application with Target Platform: Any CPU (project properties, Build). That way, it will run in x86 on x86 systems and in x64 on x64 systems. Managed code will work fine and the restriction then is that you cannot use x86-specific components. The only x86 components you should use are to run AVS and that can be in the other process. The script can run in x86, pass the computed frames through Named Pipes to the main app running in x64 that can cache those without worrying about memory.

By the way, is this an open source or closed source project?
yes, I was hoping avspmod did this too, I already suggest this https://github.com/AvsPmod/AvsPmod/issues/33

this will help a lot in avoid crash if a lot of script is opened, I and many people suffered from avspmod crashes because of 2 or 4 gb limit
__________________
See My Avisynth Stuff
real.finder is offline   Reply With Quote
Old 29th June 2016, 17:44   #25  |  Link
MysteryX
Soul Architect
 
MysteryX's Avatar
 
Join Date: Apr 2014
Posts: 2,552
You might also want to consider GitHub for hosting the source code. It's a bit harder to get started, but then it works very smoothly for managing source code.

Let me know if you need some help to move the AviSynth API code into a separate process.

Here's how I would go about moving the code into a different process.
1. I'd create a new EXE project
2. I'd change the project to x64, move the DLL that doesn't compile into the new project, and look at the compilation errors (references to that DLL)
3. Move all the code that isn't compiling into that separate project.
4. Write the code to launch the process and setup the communication interface
5. Decide on the best code structure for the inter-process communication and implement it. Process B generates frames via AviSynth and return the processed frames to Process A which stores it in a cache and display to the user.

This also creates a layer of abstraction between the UI and AviSynth; you could just as well run a VapourSynth script and return it through the same inter-process interface. You might not support it now but you might as well design it so you could easily do that.

Last edited by MysteryX; 29th June 2016 at 18:44.
MysteryX is offline   Reply With Quote
Old 29th June 2016, 21:00   #26  |  Link
shekh
Registered User
 
Join Date: Mar 2015
Posts: 768
Quote:
Originally Posted by MysteryX View Post
You might also want to consider GitHub for hosting the source code.
May I ask, why GitHub is important? Can you explain its benefits?
__________________
VirtualDub2
shekh is offline   Reply With Quote
Old 29th June 2016, 21:58   #27  |  Link
shekh
Registered User
 
Join Date: Mar 2015
Posts: 768
A suggestion: there is no need to have another process only to have more address space. You can use pagefile backed memory mapping to store frame cache in the same process that generates them. Afaik pagefile mapped memory has overhead around 10% compared to normal memory, sending it to another process through pipes should be much slower...
__________________
VirtualDub2
shekh is offline   Reply With Quote
Old 29th June 2016, 22:18   #28  |  Link
FranceBB
Broadcast Encoder
 
FranceBB's Avatar
 
Join Date: Nov 2013
Location: Metropolitan City of Milan, Italy
Posts: 2,205
@shekh... it's important because you can easily setup access to that repository, test and submit changes. For instance, since you are using C#, I assume you are using Visual Studio as well. With Visual Studio you can login to your github account, create a sharing project and work together with other people to improve your code and add new functions. Basically it will first download the whole code, then it will allow you to make some changes locally, then it will check whether someone else changed something while you were working on that code. If no one changed anything, it will let you submit your changes, otherwise, if someone did actually change the code, it will retrieve the new code, check whether there are any incompatibilities with your modifications or not and ask you again if you want to commit your changes again.

I find it very constructive and a very good way to work together with other programmers.

Last edited by FranceBB; 29th June 2016 at 22:21.
FranceBB is offline   Reply With Quote
Old 29th June 2016, 22:37   #29  |  Link
shekh
Registered User
 
Join Date: Mar 2015
Posts: 768
@FranceBB
It looks like you are confusing git and github, all you described applies the same to all git, and sourceforge` git hosting seems ok
__________________
VirtualDub2
shekh is offline   Reply With Quote
Old 29th June 2016, 22:39   #30  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
Quote:
Originally Posted by shekh View Post
A suggestion: there is no need to have another process only to have more address space. You can use pagefile backed memory mapping to store frame cache in the same process that generates them. Afaik pagefile mapped memory has overhead around 10% compared to normal memory, sending it to another process through pipes should be much slower...
Shared Memory via CreateFileMapping/MapViewOfFile is probably the easiest and fastest inter-process data transfer. Not sure if that is to what you are referring.
__________________
Groucho's Avisynth Stuff
Groucho2004 is offline   Reply With Quote
Old 29th June 2016, 23:05   #31  |  Link
shekh
Registered User
 
Join Date: Mar 2015
Posts: 768
Quote:
Originally Posted by Groucho2004 View Post
Shared Memory via CreateFileMapping/MapViewOfFile is probably the easiest and fastest inter-process data transfer. Not sure if that is to what you are referring.
It is also the easiest way to have a lot of memory in single 32bit process. This trick is hard to guess and I used it for similar purpose (frame cache inside of input driver) so decided to mention this.
__________________
VirtualDub2
shekh is offline   Reply With Quote
Old 29th June 2016, 23:21   #32  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
Quote:
Originally Posted by shekh View Post
It is also the easiest way to have a lot of memory in single 32bit process.
And another goodie - 64/32 bit inter-process data exchange.
__________________
Groucho's Avisynth Stuff
Groucho2004 is offline   Reply With Quote
Old 30th June 2016, 05:12   #33  |  Link
MysteryX
Soul Architect
 
MysteryX's Avatar
 
Join Date: Apr 2014
Posts: 2,552
Quote:
Originally Posted by Groucho2004 View Post
Shared Memory via CreateFileMapping/MapViewOfFile is probably the easiest and fastest inter-process data transfer. Not sure if that is to what you are referring.
Shared Memory File is the C++ way of doing it. C# offers many other options. You want to avoid writing to the hard drive if it's not necessary. If you do a search on C# cross-process communication, you will never see Shared Memory File ever mentioned as an option to consider.

Many of the methods work between 32-bit and 64-bit processes. You just don't want a method that involves XML serialization; you want to pass raw data, but with some structure around it. Named Pipe and Binary Remoting are two options.

Just found a performance comparison of the various communication methods.

You might as well go with WCF. It's a bit more complex to setup than Remoting but it's a more modern technology and the performance is good. Then with WCF, you have the choice between NetTcpBinding and NetNamedPipeBinding. Then all communications are Object-Oriented.

Last edited by MysteryX; 30th June 2016 at 06:09.
MysteryX is offline   Reply With Quote
Old 30th June 2016, 05:19   #34  |  Link
MysteryX
Soul Architect
 
MysteryX's Avatar
 
Join Date: Apr 2014
Posts: 2,552
Quote:
Originally Posted by shekh View Post
May I ask, why GitHub is important? Can you explain its benefits?
http://programmers.stackexchange.com...code-to-github
MysteryX is offline   Reply With Quote
Old 30th June 2016, 05:29   #35  |  Link
Sparktank
47.952fps@71.928Hz
 
Sparktank's Avatar
 
Join Date: Mar 2011
Posts: 942
As an end user, I find GitHub has less clicks to get to a download.

For some reasons, navigating SourceForge tends to be the slowest.
Even with a quad, 3.1GHz processor, 8GB ram and SSD's.
I can't see why add-ons in Firefox would hinder navigating SF, but SF has the slowest response times. After 5 folders, it becomes tedious.
If the main, big green button was the AIO download archive, I'd grab that. But in most cases, there's plenty of sub-folders that may need to be grabbed separately.

I use JDownloader to download anything as even clicking download may lead to issues (for any raw document not archived).
__________________
Win10 (x64) build 19041
NVIDIA GeForce GTX 1060 3GB (GP106) 3071MB/GDDR5 | (r435_95-4)
NTSC | DVD: R1 | BD: A
AMD Ryzen 5 2600 @3.4GHz (6c/12th, I'm on AVX2 now!)
Sparktank is offline   Reply With Quote
Old 30th June 2016, 05:56   #36  |  Link
MysteryX
Soul Architect
 
MysteryX's Avatar
 
Join Date: Apr 2014
Posts: 2,552
SourceForge is great to distribute the application as it tracks the downloads, which GitHub doesn't.

However, giving a link to a ZIP file containing the source code of the latest release doesn't support collaborative development.

With GitHub, anyone can easily navigate the source code online, view the code history, download the code, make changes and submit them to you where you can merge the changes in a single click. I would have already taken a quick look at what the AviSynth API calls look like if I could do it online, and how you structured the code and projects.

With GitHub, anyone can also see the latest developments before the next official release, which is a must for collaborative development. SourceForge also supports Git, but it's not as fast and clean as GitHub.

Also there is this to take into consideration. (although I see SourceForge has just been sold to a new company that promises to turn things around)

Last edited by MysteryX; 30th June 2016 at 06:01.
MysteryX is offline   Reply With Quote
Old 30th June 2016, 06:40   #37  |  Link
MysteryX
Soul Architect
 
MysteryX's Avatar
 
Join Date: Apr 2014
Posts: 2,552
I also would advice against writing the cache to a file. If Windows needs more memory, let it manage its paging file on the hard drive. That's what it's for.
MysteryX is offline   Reply With Quote
Old 30th June 2016, 07:21   #38  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
Quote:
Originally Posted by MysteryX View Post
Shared Memory File is the C++ way of doing it. C# offers many other options. You want to avoid writing to the hard drive if it's not necessary. If you do a search on C# cross-process communication, you will never see Shared Memory File ever mentioned as an option to consider.
It has nothing to do with C++. It's a functionality the Win32 API offers. Also, nobody said anything about writing to a file/hard drive.

Quote:
Originally Posted by MysteryX View Post
Just found a performance comparison of the various communication methods.

You might as well go with WCF. It's a bit more complex to setup than Remoting but it's a more modern technology and the performance is good. Then with WCF, you have the choice between NetTcpBinding and NetNamedPipeBinding. Then all communications are Object-Oriented.
All this is about network protocols. Is that how you want to transfer frame data?

Since you seem to be clueless about what's going on beneath your shiny Visual Studio IDE and chosen abstraction method for the Win32 API, I suggest you stop embarrassing yourself.
__________________
Groucho's Avisynth Stuff

Last edited by Groucho2004; 30th June 2016 at 07:50.
Groucho2004 is offline   Reply With Quote
Old 30th June 2016, 07:27   #39  |  Link
Groucho2004
 
Join Date: Mar 2006
Location: Barcelona
Posts: 5,034
Quote:
Originally Posted by MysteryX View Post
I also would advice against writing the cache to a file. If Windows needs more memory, let it manage its paging file on the hard drive. That's what it's for.
What are you talking about?
__________________
Groucho's Avisynth Stuff
Groucho2004 is offline   Reply With Quote
Old 30th June 2016, 09:39   #40  |  Link
MysteryX
Soul Architect
 
MysteryX's Avatar
 
Join Date: Apr 2014
Posts: 2,552
Quote:
Originally Posted by Groucho2004 View Post
Shared Memory via CreateFileMapping/MapViewOfFile
How can you have a "File" without writing to the hard drive? File = hard drive.

Quote:
Originally Posted by Groucho2004 View Post
Since you seem to be clueless about what's going on beneath your shiny Visual Studio IDE and chosen abstraction method for the Win32 API, I suggest you stop embarrassing yourself.
Quote:
NetNamedPipeBinding A secure, reliable, optimized binding that is suitable for on-machine communication between WCF applications.
That's the way communication is being done in .NET between processes. Not through a shared file but through local network communication.

Adding this level of abstraction also would make a few interesting features possible. With this level of abstraction, code-wise, you're accessing a class as if it was local and the framework handles the communication and binary transfers of data. On top of the big advantages already mentioned, it would also make those possible:
- Since the worker process handles the requests from the main thread in whichever way it wants (x64 or x86), it can process VapourSynth scripts just as easily
- Since WCF really doesn't care whether it's inter-process or inter-machines communication (just change configuration), it would even make it possible to run the script on a remove server that would return the processed frames. It could even be a server with GPU support. Maybe you wouldn't want to pay $30 or $60 a month to maintain such a server, but THAT feature would be WAY COOL!
MysteryX is offline   Reply With Quote
Reply

Tags
avisynth editor, avisynth script editor, avsedit plus, encoder gui, side by side

Thread Tools Search this Thread
Search this Thread:

Advanced Search
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off

Forum Jump


All times are GMT +1. The time now is 02:24.


Powered by vBulletin® Version 3.8.11
Copyright ©2000 - 2022, vBulletin Solutions Inc.