AVSEdit Plus - official thread

[Groucho2004]

Originally Posted by MysteryX

Didn't he say he's buffering the video to facilitate playback and that caused memory to run out?

Oh, that. I was thrown off a bit by your choice of terminology.

[MysteryX]

Originally Posted by Groucho2004

Oh, that. I was thrown off a bit by your choice of terminology.

Don't be thrown off so easily. No I wasn't talking about AviSynth's internal cache.

kagetoki, .NET and Software Architecture are my specialties. If you have any questions about how to structure the code, just ask.

Running the script in a different thread does have great advantages
1. Main app can cache without running out of memory
2. If AVS process crashes or freezes, it won't affect the main app and the main app can kill it
3. You can run both x64 and x86 scripts. Otherwise, a x86 app can only run x86 scripts and a x64 app can only run x64 scripts.

It really shouldn't be hard to implement at all. With C++, this would be a heck of a nightmare to setup cross-process communication. With C#, you have many options available to do it in a clean and simple way. C# also makes multi-threading and asynchronous operations very easy to code, make good use of it.

As for the database, I see you store it in a XML file. You might want to consider using SQLite instead. Then you can manipulate your data with LINQ and it's easy to expand your data structure as the needs evolve.

You might want to compile the application with Target Platform: Any CPU (project properties, Build). That way, it will run in x86 on x86 systems and in x64 on x64 systems. Managed code will work fine and the restriction then is that you cannot use x86-specific components. The only x86 components you should use are to run AVS and that can be in the other process. The script can run in x86, pass the computed frames through Named Pipes to the main app running in x64 that can cache those without worrying about memory.

By the way, is this an open source or closed source project?

[StainlessS]

Originally Posted by MysteryX

By the way, is this an open source or closed source project?

Full C# source via Sourceforge link in 1st post, or here:- https://sourceforge.net/projects/avs...ble/01024/src/

[real.finder]

Originally Posted by MysteryX

Don't be thrown off so easily. No I wasn't talking about AviSynth's internal cache.

kagetoki, .NET and Software Architecture are my specialties. If you have any questions about how to structure the code, just ask.

Running the script in a different thread does have great advantages
1. Main app can cache without running out of memory
2. If AVS process crashes or freezes, it won't affect the main app and the main app can kill it
3. You can run both x64 and x86 scripts. Otherwise, a x86 app can only run x86 scripts and a x64 app can only run x64 scripts.

It really shouldn't be hard to implement at all. With C++, this would be a heck of a nightmare to setup cross-process communication. With C#, you have many options available to do it in a clean and simple way. C# also makes multi-threading and asynchronous operations very easy to code, make good use of it.

As for the database, I see you store it in a XML file. You might want to consider using SQLite instead. Then you can manipulate your data with LINQ and it's easy to expand your data structure as the needs evolve.

You might want to compile the application with Target Platform: Any CPU (project properties, Build). That way, it will run in x86 on x86 systems and in x64 on x64 systems. Managed code will work fine and the restriction then is that you cannot use x86-specific components. The only x86 components you should use are to run AVS and that can be in the other process. The script can run in x86, pass the computed frames through Named Pipes to the main app running in x64 that can cache those without worrying about memory.

By the way, is this an open source or closed source project?

yes, I was hoping avspmod did this too, I already suggest this https://github.com/AvsPmod/AvsPmod/issues/33

this will help a lot in avoid crash if a lot of script is opened, I and many people suffered from avspmod crashes because of 2 or 4 gb limit

[MysteryX]

You might also want to consider GitHub for hosting the source code. It's a bit harder to get started, but then it works very smoothly for managing source code.

Let me know if you need some help to move the AviSynth API code into a separate process.

Here's how I would go about moving the code into a different process.
1. I'd create a new EXE project
2. I'd change the project to x64, move the DLL that doesn't compile into the new project, and look at the compilation errors (references to that DLL)
3. Move all the code that isn't compiling into that separate project.
4. Write the code to launch the process and setup the communication interface
5. Decide on the best code structure for the inter-process communication and implement it. Process B generates frames via AviSynth and return the processed frames to Process A which stores it in a cache and display to the user.

This also creates a layer of abstraction between the UI and AviSynth; you could just as well run a VapourSynth script and return it through the same inter-process interface. You might not support it now but you might as well design it so you could easily do that.

[shekh]

Originally Posted by MysteryX

You might also want to consider GitHub for hosting the source code.

May I ask, why GitHub is important? Can you explain its benefits?

[shekh]

A suggestion: there is no need to have another process only to have more address space. You can use pagefile backed memory mapping to store frame cache in the same process that generates them. Afaik pagefile mapped memory has overhead around 10% compared to normal memory, sending it to another process through pipes should be much slower...

[FranceBB]

@shekh... it's important because you can easily setup access to that repository, test and submit changes. For instance, since you are using C#, I assume you are using Visual Studio as well. With Visual Studio you can login to your github account, create a sharing project and work together with other people to improve your code and add new functions. Basically it will first download the whole code, then it will allow you to make some changes locally, then it will check whether someone else changed something while you were working on that code. If no one changed anything, it will let you submit your changes, otherwise, if someone did actually change the code, it will retrieve the new code, check whether there are any incompatibilities with your modifications or not and ask you again if you want to commit your changes again.

I find it very constructive and a very good way to work together with other programmers.

[shekh]

@FranceBB
It looks like you are confusing git and github, all you described applies the same to all git, and sourceforge` git hosting seems ok

[Groucho2004]

Originally Posted by shekh

A suggestion: there is no need to have another process only to have more address space. You can use pagefile backed memory mapping to store frame cache in the same process that generates them. Afaik pagefile mapped memory has overhead around 10% compared to normal memory, sending it to another process through pipes should be much slower...

Shared Memory via CreateFileMapping/MapViewOfFile is probably the easiest and fastest inter-process data transfer. Not sure if that is to what you are referring.

[shekh]

Originally Posted by Groucho2004

Shared Memory via CreateFileMapping/MapViewOfFile is probably the easiest and fastest inter-process data transfer. Not sure if that is to what you are referring.

It is also the easiest way to have a lot of memory in single 32bit process. This trick is hard to guess and I used it for similar purpose (frame cache inside of input driver) so decided to mention this.

[Groucho2004]

Originally Posted by shekh

It is also the easiest way to have a lot of memory in single 32bit process.

And another goodie - 64/32 bit inter-process data exchange.

[MysteryX]

Originally Posted by Groucho2004

Shared Memory via CreateFileMapping/MapViewOfFile is probably the easiest and fastest inter-process data transfer. Not sure if that is to what you are referring.

Shared Memory File is the C++ way of doing it. C# offers many other options. You want to avoid writing to the hard drive if it's not necessary. If you do a search on C# cross-process communication, you will never see Shared Memory File ever mentioned as an option to consider.

Many of the methods work between 32-bit and 64-bit processes. You just don't want a method that involves XML serialization; you want to pass raw data, but with some structure around it. Named Pipe and Binary Remoting are two options.

Just found a performance comparison of the various communication methods.

You might as well go with WCF. It's a bit more complex to setup than Remoting but it's a more modern technology and the performance is good. Then with WCF, you have the choice between NetTcpBinding and NetNamedPipeBinding. Then all communications are Object-Oriented.

[MysteryX]

Originally Posted by shekh

May I ask, why GitHub is important? Can you explain its benefits?

http://programmers.stackexchange.com...code-to-github

[Sparktank]

As an end user, I find GitHub has less clicks to get to a download.

For some reasons, navigating SourceForge tends to be the slowest.
Even with a quad, 3.1GHz processor, 8GB ram and SSD's.
I can't see why add-ons in Firefox would hinder navigating SF, but SF has the slowest response times. After 5 folders, it becomes tedious.
If the main, big green button was the AIO download archive, I'd grab that. But in most cases, there's plenty of sub-folders that may need to be grabbed separately.

I use JDownloader to download anything as even clicking download may lead to issues (for any raw document not archived).

[MysteryX]

SourceForge is great to distribute the application as it tracks the downloads, which GitHub doesn't.

However, giving a link to a ZIP file containing the source code of the latest release doesn't support collaborative development.

With GitHub, anyone can easily navigate the source code online, view the code history, download the code, make changes and submit them to you where you can merge the changes in a single click. I would have already taken a quick look at what the AviSynth API calls look like if I could do it online, and how you structured the code and projects.

With GitHub, anyone can also see the latest developments before the next official release, which is a must for collaborative development. SourceForge also supports Git, but it's not as fast and clean as GitHub.

Also there is this to take into consideration. (although I see SourceForge has just been sold to a new company that promises to turn things around)

[MysteryX]

I also would advice against writing the cache to a file. If Windows needs more memory, let it manage its paging file on the hard drive. That's what it's for.

[Groucho2004]

Originally Posted by MysteryX

Shared Memory File is the C++ way of doing it. C# offers many other options. You want to avoid writing to the hard drive if it's not necessary. If you do a search on C# cross-process communication, you will never see Shared Memory File ever mentioned as an option to consider.

It has nothing to do with C++. It's a functionality the Win32 API offers. Also, nobody said anything about writing to a file/hard drive.

Originally Posted by MysteryX

Just found a performance comparison of the various communication methods.

You might as well go with WCF. It's a bit more complex to setup than Remoting but it's a more modern technology and the performance is good. Then with WCF, you have the choice between NetTcpBinding and NetNamedPipeBinding. Then all communications are Object-Oriented.

All this is about network protocols. Is that how you want to transfer frame data?

Since you seem to be clueless about what's going on beneath your shiny Visual Studio IDE and chosen abstraction method for the Win32 API, I suggest you stop embarrassing yourself.

[Groucho2004]

Originally Posted by MysteryX

I also would advice against writing the cache to a file. If Windows needs more memory, let it manage its paging file on the hard drive. That's what it's for.

What are you talking about?

[MysteryX]

Originally Posted by Groucho2004

Shared Memory via CreateFileMapping/MapViewOfFile

How can you have a "File" without writing to the hard drive? File = hard drive.

Originally Posted by Groucho2004

Since you seem to be clueless about what's going on beneath your shiny Visual Studio IDE and chosen abstraction method for the Win32 API, I suggest you stop embarrassing yourself.

NetNamedPipeBinding A secure, reliable, optimized binding that is suitable for on-machine communication between WCF applications.

That's the way communication is being done in .NET between processes. Not through a shared file but through local network communication.

Adding this level of abstraction also would make a few interesting features possible. With this level of abstraction, code-wise, you're accessing a class as if it was local and the framework handles the communication and binary transfers of data. On top of the big advantages already mentioned, it would also make those possible:
- Since the worker process handles the requests from the main thread in whichever way it wants (x64 or x86), it can process VapourSynth scripts just as easily
- Since WCF really doesn't care whether it's inter-process or inter-machines communication (just change configuration), it would even make it possible to run the script on a remove server that would return the processed frames. It could even be a server with GPU support. Maybe you wouldn't want to pay $30 or $60 a month to maintain such a server, but THAT feature would be WAY COOL!