Avisynth virtual file system

[Adub]

So, wait a second. Could I use this to create an avs script on one computer, have another linux computer connect to that script via network share, and then use my linux computer to encode that script? Is all avs processing done on the host computer? Like am I feeding raw data to my linux computer?

I am thinking that I might be able to apply this so that I run all of my scripting on an Avisynth and AVFS enabled XP machine, and then via a gigabit network, send the raw data a 64-bit Gentoo install were I encode the final output using mplayer with x264.

[joe48]

I am thinking that I might be able to apply this so that I run all of my scripting on an Avisynth and AVFS enabled XP machine, and then via a gigabit network, send the raw data a 64-bit Gentoo install were I encode the final output using mplayer with x264.


This should work. Post your results if you try it. Most interesting is how much CPU utilization you are able to get on each machine. Ideally at least one of the machines would have one CPU core fully utilized.

[Adub]

Cool. I can't do it yet, but I might be able to get it working this summer. If I do (or don't) get it working, I will be sure to post back.

[Adub]

Okay!! I got it working!! Which is good. However, it's disappointingly slow.

Here is what I did:

Log into windows, and then share my working folder.

I mount that shared folder in my linux machine with "sudo mount -t smbfs //ip.add.res.s/Videos /media/share"

I can browse around the shared folder and play the files using mplayer. I can even encode them!! Now, here is where the example comes in.

If I take my example file, and stream it directly to x264 via mencoder, I can get around 122fps on the first pass and ~12.18 fps on the second pass.

However, if I use AVFS, I get around 15fps on the first pass. I didn't even test the second pass.

The script was very simple, just a basic "Directshowsource("example.mkv")". I don't know if there is something I can do to improve it, but right now it's too slow for me to use in my encoding pipeline.

Edit: And as to CPU utilization, I didn't get above 20 % on either machine. The one running AVFS barely went above 10.

[Shinigami-Sama]

betcha your network maxed out
SMB is stupidly inefficient...

[Adub]

But, that makes no sense. Using the same network and same mounting techniques, I got 122 fps without using AVFS to 15 fps using AVFS. That makes little sense.

Unless.....Oh!!! Is it because I am sending UNCOMPRESSED? And that would result in bigger frame sizes, thus resulting in the decreased network speed, thus the decreased encoding speed. Is there a better alternative to SMB?

Note: This was on a 100mb network. Maybe if I can't fix it with an alternative to SMB, would an upgrade to 1000mb network work?

[mikeytown2]

Okay!! I got it working!! Which is good. However, it's disappointingly slow.

Since the network seems to be the bottleneck, make AVS the bottleneck. What i mean is, this would be useful when the AVS script takes 100% cpu, because then the network would no longer slow you down. So the idea of using this would make since when using monster scripts.

Also for alt to SMB i found this thread
http://www.neowin.net/forum/index.php?showtopic=308559

Edit: look at the taskman networking tab to make sure the network is maxing out

Edit2: If your looking at 10GB net (http://www.google.com/products?q=dual-port+10Gb+network&oe=utf-8&scoring=p), its about 400 bucks per computer...

[joe48]

However, if I use AVFS, I get around 15fps on the first pass. I didn't even test the second pass.


RGB24 AVI at 720x480 is ~1MB per frame. 80% of 100Mbit network is ~8fps. 30% of 1Gbit network is ~30fps. The frame rates double for YV12. Depending on your network, 15fps may be all you can hope for. The 2nd pass may run at the same speed though.
(Hope I got the math right)

[Shinigami-Sama]

RGB24 AVI at 720x480 is ~1MB per frame. 80% of 100Mbit network is ~8fps. 30% of 1Gbit network is ~30fps. The frame rates double for YV12. Depending on your network, 15fps may be all you can hope for. The 2nd pass may run at the same speed though.
(Hope I got the math right)

maybe passing it off to an i-frame only lossless codec could speed it up?
huffman maybe
should double or so the network access
shouldn't be to hard to do either

[mikeytown2]

maybe passing it off to an i-frame only lossless codec could speed it up?
huffman maybe
should double or so the network access
shouldn't be to hard to do either

VLC (http://www.videolan.org/) sounds like it could do the job. Speaking of VLC, could it accept avs files for input (http://forum.videolan.org/viewtopic.php?f=14&t=38283)?

[Adub]

Okay, so it IS my network. Let me see if I can find a gigabit card for my laptop.

[Shinigami-Sama]

actually
cold the fake AVI file be passed off to a http server? I don't see why not
you could easily have a script pass it off to a lossless encoder
though I guess that might not work for some applications...

[IanB]

People need to consider the data being virtualized here is big, really big. A RGB24 1920x1080 24fps progressive stream is 149,299,200 bytes per second. Even a YV12 (4:2:0) 720x480 60i is still 15,552,000 bytes per second. This is 24% more than the theoretical maximum transfer of a 100Mb/s ethernet (probably more than twice the practical throughput). These numbers are even a small chunk out of a 800MHz FSB memory channel at around 6,400,000,000 bytes per second.

[joe48]

Lossless compression in AVFS is not a solution to network bandwidth problems. AVFS would need to preprocess (compress) all frames to figure out file layout and indexing. It would make more sense to skip AVFS and losslessly encode to a real AVI file.

Serving over a network is still useful when the CPU overhead of the script and encoder are limiting throughput below what the network can handle.

[Adub]

Yeah, this is making more and more sense. Looks like I have a nice project to work on over the summer! I just need to setup a decent gigabit network, and look at my filter chain to see if I can change the bottleneck to be the avs, not my network.

[Zarxrax]

I've found a very strange bug. I have a folder full of a few hundred avisynth scripts, and all of them contain the line
AssumeFPS("ntsc_double")
I was going through mounting each one, one at a time, and i noticed that some of them would mount fine, but when i tried to mount others, it would crash. On these that failed to mount, I discovered that I could get them to mount by simply removing the AssumeFPS line. That's really strange though, since all of my scripts have that line, and some of them work. I've determined that it is this specific line that it causing it to crash though, in many of my scripts.

[joe48]

I've found a very strange bug. I have a folder full of a few hundred avisynth scripts, and all of them contain the line
AssumeFPS("ntsc_double")


If you can get me a sample exhibiting the problem I can look at it. Send me a PM.

[morsa]

Could something simillar to "FireHose" from Cinelerra be modified for our uses?

[Quark.Fusion]

About compression on network: did someone tried open share over compressed vpn link?

[Quark.Fusion]

Also about networking with avisynth: avisynth already have networking functionality with TcpServer(), will it work with mounted file?

What about adding multithreaded Lagarith compression for networking to AVFS? For user this can look like adding two variables to avs file with port number and compression method (like in TcpServer), on other side user will mount special file, that will re-create avi file in virtual filesystem (like using TcpSource). Of course all functionality already here (you can just mount TcpSource() avs file), but it is with ugly code (i was read that somewhere) and without lagarith compression.

[joe48]

I've found a very strange bug. I have a folder full of a few hundred avisynth scripts, and all of them contain the line
AssumeFPS("ntsc_double")

If you can get me a sample exhibiting the problem I can look at it. Send me a PM.


The bug Zarxrax found is a divide-by-zero with clips that are less than 1 second long. I will have the fix in the next release.

[joe48]

What about adding multithreaded Lagarith compression for networking to AVFS? For user this can look like adding two variables to avs file with port number and compression method (like in TcpServer), on other side user will mount special file, that will

TcpSource and AVFS should interoperate fine.

I am not very familiar with TcpSource. It should not be necessary to know the compressed size of each frame in advance, so lossless compression could be implemented...

[tally3]

TGA sequences would be really useful!
http://www.organicbit.com/closecombat/formats/tga.html

[thommyfilm]

hello! sounds like very good work. one (maybe dumb) question: you say you would add output file format options. would it be possible to show an mpeg4 h264 file on the filesystem instead of a avi file? this should happen without reencoding. the input files are already mp4 h264 files.

[krieger2005]

You can't do it without reencoding. Every change, which you made with an avisynth-script to your video must be reencoded. This is a simple Law:
-> No change -> No Reencoding -> You don't need to use avisynth
-> change (in anyway) -> need reencoding

[mikeytown2]

@thommyfilm, thats what MeGUI is for ;)

[Adub]

Uh, guys, I think he is asking if support for a "fake" mp4 front could be added in addition to the "fake" avi front. I don't know why this could be, as everything supports avi, while mp4 is much more limited.

[IanB]

Also would programs that accept MP4 containers handle uncompressed video and audio data. I suspect not.

[joe48]

There may be some performance changes when AVFS is used with the latest PFM Audit Package (build 048). I changed the caching in the file system driver to use 256k blocks instead of 48k. I did not test this against AVFS, probably should have...

I have some optimizations in the works for Pismo File Mount that will improve AVFS performance. Mainly there will be buffering improvements to eliminate extra data copying. It will be a while before I get this out, probably January.

Joe L.

[Adub]

Great!! I look forward to the next release!

Thanks for your hard work joe48.

[hanfrunz]

I have another wish. Is it possible to mount multiple .avs to one virtual folder or drive? So lets say i have clip1.avs, clip2.avs and i like to have \clips\clip1.avi and \clips\clip2.avi? So i could import two files via drag and drop to my encoder.

that would be realy helpful.

regards,
hanfrunz

[joe48]

I have another wish. Is it possible to mount multiple .avs to one virtual folder or drive? So lets say i have clip1.avs, clip2.avs and i like to have \clips\clip1.avi and \clips\clip2.avi?


On vista you might be able to fake this using a file search.

If the \clips\ folder is a real folder then this would be difficult. On vista it could maybe be done by having avfs create and delete NTFS symbolic links.

If the \clips\ folder is a virtual folder, then it would require more UI, probably an .avs file list mount file. It also would result in multiple avisynth instances running in a single process, which might be problematic.

[hanfrunz]

On vista you might be able to fake this using a file search.

filesearch is a very good idea and should work on xp as well, thanks!

[mikeytown2]

Any chance of adding YUV output as well as RGB?
http://forum.doom9.org/showthread.php?p=1218241#post1218241

[IanB]

@mikeytown2,

:o :o It already does. :o :o

AVFS outputs the exit format of your Avisynth script, RGB24, RGB32, YUY2 or YV12

[paulvdb]

I can't get this to work. I download and installed Pismo and then registered the avfs dll. When I mount an avs file it creates a new folder with that same avs file in it and a file called error.log which just context the text "(null)". Should I do anything else to get it to work? I've tried this with several avs files including the sample.avs that came with AVFS. I'm using Windows XP SP2 with all the latest updates.

[paulvdb]

I managed to solve my problem by upgrading avisynth. Apparently AVFS doesn't work with avisynth 2.56.

[joe48]

I released a new version of Pismo File Mount Audit Package earlier this month, build 050.

This new version has some substantial changes to buffering and caching to eliminate unnecessary memory copying. Depending on data sources and AVS scripts, there might be some performance increase using AVFS on build 050 and later. No changes to AVFS itself were necessary, so the AVFS build has not been updated.

Build 050 also uses 128kb sized cache buffers in the driver, instead of 256k in build 047-048, and 48k in build 046 and earlier. This only affects client applications that do not do unbuffered I/O.

Joe L.

[Quant]

And how about BoxedApp SDK (http://boxedapp.com/). I heared, that this SDK is for emulating file system and registry too? Does anybody use this?

[a451guy451]

I love this thing, but it only lets me host virtual files (that my Mac can see) on shared folders physically on the host windoz machine. Is there a way to get files on independent network storage to work, because while SD sourced files work great HD stuff chokes on my ethernet card? Right now, I technically can make virtual files on network storage, but they are only accessible as such via my host machine which defeats the purpose. Or perhaps this is impossible and I need to go buy a 10gbps ethernet card... Thanks for any input!

[a451guy451]

Oops. I think I'm having a FSB bottleneck, not ethernet. Just swapped my penrin host machine with an i7 and it dramatically improved.

Oddly enough, if I connect to my virtual folder from any PC on my network, everything works great. It seems my Mac is having a personal problem (or perhaps quicktime is specifically). Does anybody else make virtualfiles using HD sources and use them on their Mac? I'm low on apple expertise, and am not really sure how to troubleshoot the system for the bottleneck there.

[Dreadkillz]

Hi, I would really love to use this, but whenever I so much as select the avi file from my script, my folder explorer would just freeze. I checked that pfmhost also grows in memory usage by a lot. Loading the virtual file is A LOT slower than loading the actual script file for me. Also, the heavier that my filters are, the slower it is to load. BTW, I have an XP with AMD 3600 x2

[joe48]

Hi, I would really love to use this, but whenever I so much as select the avi file from my script, my folder explorer would just freeze. I checked that pfmhost also grows in memory usage by a lot. Loading the virtual file is A LOT slower than loading the actual script file for me. Also, the heavier that my filters are, the slower it is to load. BTW, I have an XP with AMD 3600 x2

Performance is often slower with AVFS than a frame served or VFW accessed AVS, but it depends on the script and on the program accessing the virtual AVI file. The problem is not that AVFS adds any significant overhead to rendering, because it doesn't (there are file systems built on PFM doing hundreds of MBs per second sustained throughput). The problem is that the programs accessing the virtual AVI files often issue reads that require frames to be rendered that the application does not use.

With the limited types of scripts I use, the performance issue I see is a delay when opening the virtual AVI. This is caused by sloppy reads of the AVI file headers and indexes inadvertently causing unneeded frames to be rendered.

There may be some things that can be done with the tunable AVFS parameters in the script to improve performance. Also, someone with some C/C++ coding experience, who is seeing performance issues, could experiment with caching and/or padding in a custom avfs.dll.

[IanB]

... but whenever I so much as select the avi file from my script, my folder explorer would just freeze. I checked that pfmhost also grows in memory usage by a lot. ...
Various tools and explorer plugins like to 'investigate' .AVI files. And then there are virus scanners and security products. It is crucial these never be allowed to read through the contents of the pseudo .AVI files. If they do then the pseudo contents has to be rendered. If it takes 2 hours to render the full Avisynth script, then it will take 2 hours to read through the pseudo .AVI file.

Joe, does PFM have knowledge of the "Application name" when a pseudo file access happens? Could various applications be black listed and the returned contents just be 1 big zero fill. Or maybe a white list so only say VirtualDub.exe, X264.exe and Vegas*.exe get to see the rendered pseudo contents.
... The problem is that the programs accessing the virtual AVI files often issue reads that require frames to be rendered that the application does not use.

... This is caused by sloppy reads of the AVI file headers and indexes inadvertently causing unneeded frames to be rendered. ...
In the original implementation, I went for up tight maximum packing in the pseudo AVI file. On reflection perhaps a sparse highly padded approach would have been better.

In an AVI structure 'JUNK' segments can be freely used to align/pad other data segments, this is normally done when capturing uncompressed video to match the clustering alignment of physical disks and maximise performance.

It would be trivial to have the frame data aligned on some nice round offset (say 64K or more) and add a 'JUNK' padding/guard block between each frame chunk. The data contents in the 'JUNK' blocks doesn't even matter, so no actual memory transfer/zero fill needs to occur, further boosting performance.

I assume the marshalling code is happy if no actual data is written to the application's buffer.
_____________________________________

Case study :- On mapped Linux systems the /dev/zero driver can apparently achieve throughputs of 100's of GB/s, way in excess of the actual memory bandwidth of the host CPU. This is achieved by zero filling 1 physical memory page and mapping it repeatedly onto the users buffer. E.g. App does a 1Mb _read of /dev/zero, on a modern CPU one might expect this to take 40 microseconds (25GB/s), profile the _read call and it only takes 5 microseconds (200Gb/s). Woooohoooo!

[joe48]

Various tools and explorer plugins like to 'investigate' .AVI files. And then there are virus scanners and security products. It is crucial these never be allowed to read through the contents of the pseudo .AVI files.

.....

Joe, does PFM have knowledge of the "Application name" when a pseudo file access happens? Could various applications be black listed and the returned contents just be 1 big zero fill. Or maybe a white list so only say VirtualDub.exe, X264.exe and Vegas*.exe get to see the rendered pseudo contents.


No current mechanisms exist in PFM to allow the formatter to change behavior based on the process name. If it were supported, it would only be available during open, not at read.

But, I really do not like this sort of a solution. It looks like a quick fix, but it has a lot of drawbacks.

Examples:
1) Explorer related I/O, such as various shell extensions, is going to happen in the open dialog in applications in the white list.
2) Some of the anti-virus products and system extensions open files and do I/O in the context of the triggering process (this is true of prefetch as well).
3) Some apps/plugins may do work out-of-process, so the user has to track down the name of processes they did not run and add them to the white list.
4) Dllhost.exe, rundll32.exe, and the like.

One thing that should help is to set the FILE_ATTRIBUTE_OFFLINE flag on the virtual media files. This is a trivial change to avfs.dll. I would have done this originally, but it was not supported two years ago when we first put AVFS together. This flag will convince explorer to leave the file alone, only show default icon, etc.. Many 3rd party shell extensions and applications also honor this flag, otherwise they are unusable in enterprise environments that use any sort of HSM. The shell extensions that do not check this flag should be bugged back to the developer.

The anti-virus issues can be handled (usually) with exclusions. Well, if the anti-virus product does not support user exclusions then the user might want to look for another vendor.

In the original implementation, I went for up tight maximum packing in the pseudo AVI file. On reflection perhaps a sparse highly padded approach would have been better.

...

It would be trivial to have the frame data aligned on some nice round offset (say 64K or more) and add a 'JUNK' padding/guard block between each frame chunk. The data contents in the 'JUNK' blocks doesn't even matter, so no actual memory transfer/zero fill needs to occur, further boosting performance.


I can not recall if there were any compat issues with the padding between frame and audio chunks. There is the existing AVFS_AVI_NoInterleave option to disable interleaving the audio and video chunks. Some more padding options would probably be useful.

The below thread indicates the no-interleave option solves some performance issues. I am suspicious about the comment that it introduces audio sync issues, but this is the sort of compat trouble that playing with the index and with padding is likely to cause.
http://www.dvinfo.net/forum/canon-eos-5d-mk-ii-hd/140473-intermediate-codecs-windows-users-2.html#post990122

[Robert Martens]

Talk about a small world; after years of lurking, I registered an account here a few weeks ago to post about an Avisynth helper script I'd developed. I ended up revising it practically every other day, though, and thought it best to wait until I'd settled on something more stable before bringing it up here, so I have yet to get around to that (maybe later tonight, actually), but I was just browsing the boards and stumbled across this.

As evidenced by my username, I'm the same guy from the aforementioned DV Info Net thread. It's been over a year since I last experimented with the AVFS as it applies to 5D Mark II Quicktime files, so I'm a little fuzzy on the details, but yes, I do remember that while there was a performance improvement brought about by AVFS_AVI_NoInterleave, we subsequently ran into an issue with A/V sync. Barely being able to "Hello, World!" my way out of a wet paper bag, I was and still am in no position to diagnose the cause, and the simplest solution was just avoiding that option. Ultimately I believe most people started using a Cineform transcoding solution for their 5D (and now 7D and T2i) footage for use in their NLEs.

I still have the sample clip I was using for tests in that DVi thread (on the same system, no less, with the same processor, RAM and hard drives, though now I'm using Avisynth 2.5.8 MT), and I'm happy to help if there's anything I can do as far as testing. Or, at least, shedding some light on that particular discussion. It's been awhile, but I'm sure I could remember the details if you have any questions.

[joe48]

I do remember that while there was a performance improvement brought about by AVFS_AVI_NoInterleave, we subsequently ran into an issue with A/V sync.
....
and I'm happy to help if there's anything I can do as far as testing. Or, at least, shedding some light on that particular discussion. It's been awhile, but I'm sure I could remember the details if you have any questions.

If you get a chance, could you rerun the test with and without the no-interleave option and verify the audio sync issue comes and goes?

[Robert Martens]

Turns out more had changed than I remembered, in that I had been using the CoreAVC decoder and loading the fake AVI files into a trial copy of Vegas Pro. I've downloaded Vegas too many times now (I've needed it more than once to help someone diagnose a problem; it now tells me the trial period has expired, so I can't register the download), and pathetic as it may seem I can't throw any money at CoreAVC for the moment.

I have nonetheless confirmed the behavior in question by other means. I have ffdshow-tryouts version 3154 installed, and I have the H.264/AVC "decoder" dropdown set to ffmpeg-mt for both DShow and VfW, since I find that performs best on my system.

I created a simple script, AVFSsynctest.avs, containing these lines:

AVFS_AVI_NoInterleave=false
DirectShowSource("C:\Downloads\mvi_1419.mov")
ConvertToRGB24(matrix="PC.709")

The source clip doesn't appear to be in the same place as when I grabbed it, but that Vimeo user referenced in the DVi thread has made the clip available here: http://drop.io/larktav

I swapped NoInterleave between true and false, and commented out the ConvertToRGB line for a couple of tests. The script was mounted with the latest PFM/AVFS, the files were opened in Liquid 7.2, combustion 3.0.4, and MediaInfo, and the script was explicitly Unmounted between each change. Results were as follows:

1.) NoInterleave=false->DirectShowSource->ConvertToRGB24->Liquid 7.2

On a 1080/30p timeline (exactly 30p, not 29.97), everything displays as if it's fine; the audio in the AVI file shows up as being the same duration as the video, as do the two WAV files generated (AVFSsynctest.wav and AVFSsynctest.00.wav). I'm inclined to believe this is a quirk of the NLE, however, thanks to tests 3 through 8.

2.) NoInterleave=true->DirectShowSource->ConvertToRGB24->Liquid 7.2

Same as 1; everything looks fine, durations all appear to match.

3.) NoInterleave=false->DirectShowSource->combustion 3.0.4

Here's where the first sign of trouble begins. The video track is reported as 491 frames, but audio is reported as 490 whether I load it from the AVI container or either of the WAVs.

4.) NoInterleave=true->DirectShowSource->combustion 3.0.4

Even stranger; the durations behave as in test 3, but trying to load the AVI as an audio source causes only the last ten frames or so to display a waveform in the combustion interface; further, the audio track is silent even during those ten frames. The WAVs as audio sources had the 490 duration, but played back without issue.

5.) NoInterleave=false->DirectShowSource->ConvertToRGB24->combustion 3.0.4

For tests 3 and 4 I had initially removed the ConvertToRGB line, as combustion doesn't require it, but thinking it might have some impact on my results I restored the line and tried again. Results were identical to test 3.

6.) NoInterleave=true->DirectShowSource->ConvertToRGB24->combustion 3.0.4

Same as number 4.

7.) NoInterleave=false->DirectShowSource->ConvertToRGB24->MediaInfo

Finally, I loaded the phantom files in MediaInfo 0.7.19 to see what it could tell me, and found that in this case the AVI video track showed a duration of 16s 366ms, and the audio a duration of 16s 333ms. The external files, AVFSsynctest.wav and AVFSsynctest.00.wav, were also both 16s 333ms.

8.) NoInterleave=true->DirectShowSource->ConvertToRGB24->MediaInfo

Changing only the NoInterleave option, this time I found the AVI video track was its expected 16s 366ms, but the audio packed into the AVI had shrunk to 15s 867ms. AVFSsynctest.wav and AVFSsynctest.00.wav, though, were both 16s 333ms, as I'd seen in the other examinations.

All of this is in line with what we experienced in that thread; I was unclear at first, but as soon as I saw those numbers in MediaInfo I remembered running into the same audio-shrink last time.

[IanB]

Joe, Seems you have a bug, you forget to put back the 0.5 secs of preload when not in interleave mode.

[joe48]

@Robert, I sent you a PM with a link to a test build of AVFS. It would be great if you could give it a look.