Avisynth virtual file system

[joe48]

How much interest is there in DPX frame stack support for AVFS?

This would work as one or more "frames" folders under the AVS script mount point, each containing 1000's of "NNNN.DPX" files.

I think I could code this without too much effort, but I am not setup to work out the required features or properly test. Some help from someone who works with a variety of related tools would be needed.

[kolak]

It would be great, but MOV would be awesome :)

I can help with development.

Some pro tools for DPX reading/writing:

Assimilate Scratch Play- free advanced player
Davinci Resolve - read/write
http://pomfort.com/dpxheadereditor/ - headers viewer, editor
http://www.tweaksoftware.com/downloads/tweak-software-license-terms-and-conditions - advanced player (trial)
Nuke - read/write has personal learning mode, so no license needed
http://www.gluetools.com/downloads_dpx.html - DPX QT component, has trial mode

The most important is the way how data is packed- there are few modes, but looks like most people use 1 of them (I think method A) due to better compatibility with GPUs. There is also now DPX ver. 2.
Mediainfo can print very detailed info about DPX.

[joe48]

I may get some time next week to do some coding while I am at a conference. If so I will work on the general frame stack style output support, and go from there.

If you want to help with dev, setup for building AVFS with VC6 is pretty straightforward. All the code should be in the release package, or you can get it from the sourceforge Avisynth CVS repository.

[kolak]

I can help with testing, practical "approach", knowledge around DPX usage etc, but not really with coding part :(

Other than DPX maybe you can have a look at EXRs, as this is forming as a new "standard" for high-end work. DPX and EXRs have much more sense with vapoursynth, which does all pixel formats up to 32bit float.
VS already has your code implemented with some additional features like v210 support, which was requested by me and swiftly implemented by Myrsloik :)

I have a general/big idea about virtual video file system, but unfortunately no skills to make it "real".

[tyee]

Just tried the latest version. I suppose I have to double click the mounted .avs file within the c:\volumes folder? When I do this I see the avi file and the original avs file. Can't seem to play it. Do I drag this avi file into a media player to test it? I tried mpc-hc and it won't load. If I go back one folder directory in my file manager and then go back into c:\volumes and double click the avs file again my file manager hangs because mpc-hc is still trying to load it. Just found out that mpc-hc won't close in task manager until I close cmd window of avfs.exe. My avi file size is 200GB and mpc-hc is still opening it after 2 minutes with nothing showing. Am I doing something wrong? Your sample.avs works fine. I'll try some of your tips

[joe48]

... double click the avs file again my file manager hangs. Just found out that mpc-hc won't close in task manager until I close cmd window of avfs.exe ...

The c:\volumes\xxxxx.avs is not your AVS file, it is a temporary mount point to expose the AVFS virtual volume created from your script, like a virtual folder, not your script file. I hope you are not copying your AVS file into c:\volumes .

Make sure your AVS file opens and functions outside of AVFS, for example using Virtualdub. Next make sure you can mount a simple sample AVS file, such as the sample.avs file included in the AVFS package.

[tyee]

Hi Joe
Thanks for the fast reply. OK, still can't open it. The mounting goes well and I can see my mounted .avs file within the c:\volumes folder. So how do I load this into mpc-hc?

I have been double clicking the mounted avs file which shows me the large avi file. I then drag and drop that large avi to mpc-hc but it never finishes loading. My avs itself loads and plays fine though. Note - I am using avisynth MT in my script, does that matter?

Update - I disabled some lines in my avs file, namely the MT portion, and the QTGMC portion. Looks like just enabling MT works ok, loads and plays fine. When I enable the QTGMC filter, this causes the hang and it never loads to play.

[joe48]

... Looks like just enabling MT works ok, loads and plays fine. When I enable the QTGMC filter, this causes the hang and it never loads to play.

I don't know what issues there may or may not be when using AVFS with MT, or with specific filters and MT. Probably need the filter developer to look at what is going on in the avfs.exe process when it is stuck.

Note that AVFS is compiled against an Avisynth 2.5.x API header, which predates the MT branch/changes. Possible the filter you are using is just not compatible with AVFS unless AVFS is rebuilt with a different Avisynth API version.

[Overdrive80]

Hi Joe
Thanks for the fast reply. OK, still can't open it. The mounting goes well and I can see my mounted .avs file within the c:\volumes folder. So how do I load this into mpc-hc?


For example, copy avfs.exe in avisynth 2.5 plugins folder, create you cmd file, for example:

"C:\Program Files (x86)\AviSynth 2.5\Plugins\avfs.exe" "Path\Premiere.avs"

You can found in C:\Volume folder you virtual file. You should too install last version of pismo (build 175). Maybe you have remove old avfs.dll for discarding conflicts, I think.

EDIT: To Lost shell integration is a enormous lost.

[kolak]

Update - I disabled some lines in my avs file, namely the MT portion, and the QTGMC portion. Looks like just enabling MT works ok, loads and plays fine. When I enable the QTGMC filter, this causes the hang and it never loads to play.

QTGMC can be very problematic.
Read this:

http://forum.doom9.org/showthread.php?p=1609171#post1609171

[Sparktank]

QTGMC can be very problematic.
Read this:

http://forum.doom9.org/showthread.php?p=1609171#post1609171

pfmhost should be LAA by now.
I think the only thing of interest is to use ditributor() for setmt scripts.

[Bruce/Fl]

I gave up on VFAPI, (see this thread (http://forum.doom9.org/showthread.php?t=172553)) at least for now, and am trying AVFS. It installed fine, as did the PFM program, but it won't open an avs file, either by clicking on the executable, using the PFM mount option, or by using the cmd line. I saw some references here about "registering" the avfs.dll. I'm not familiar with the concept, but thought I'd find the dll and look, and I can't find the avfs.dll.

Anybody have an idea what I'm doing wrong?

I'm on a Windows 7 Pro 64 bit.

Thank you,

Bruce

[Bruce/Fl]

There is no avfs.dll now in the latest version. Have you done what the readme says?

Yes, I put a copy of the avfs.exe in the windows folder. I then tried the "avfs somescriptfile.avs" after the prompt it goes to "Press CTRL+C to exit." And I can't enter anything else in it at that point.

When I tried the "start avfs somescriptfile.avs" after the prompt, another cmd box opens this time, labeled simply C:|Windows\avfs.exe, with "Press CTRL+C to exit" in it. And I can't enter anything else in it at that point again.

I rarely if ever run cmd commands, so maybe I am missing something obvious. I am interpretating "somescriptfile" as the name of my .avs file, and simply inserting that name there. Or does it need more info like as in the location of the file? So in this case, I am simply writing "avfs movtest1.avs" after the "C:\Users\Bruce>" prompt and hitting enter. Is that the issue?

[TheFluff]

Yes, if the file isn't in C:\users\bruce you need to specify the full path to it, not just the filename. So, for example
start avfs "c:\some\folder\somescriptfile.avs"
The quotes are important (unless you don't have any spaces or other funny characters in the path but I wouldn't take any chances, just put quotes around the path always and you're safe)

[Bruce/Fl]

Well...you don't need to enter anything after the script is mounted (when you see "Press CTRL+C to exit." appears). Like the readme says (in the Operation section), the script output is presented as various *.avi and *.wav files in the C:\Volumes\somescriptfile.avs folder. Now you can use your favorite application to open that avi file as a normal video for processing then.

Thank you both, I got past that point now, and I have a set of files in my "volumes" folder.

Now, when I try to open them in Vegas (Pro 13) I get the error message: "Warning: An error occerrred while opening one or more files. An error occurred while opening a codec"

Did I miss loading up something? The file is in .avi format, never had a problem opening one of those before.

[Reel.Deel]

I'm still using AVFS v1.0.0.5 (https://web.archive.org/web/20140707231925/http://www.turtlewar.org/avfs/), the latest version limited shell integration (it was very handy :().

----

@joe48

Is it OK to just AVFS v1.0.0.5 with the latest version of Pismo File Mount? I'm using Pismo File Mount build 166 right now but I want to make sure before I upgrade to build 178.

[Iron_Mike]

Hi,

is it possible to route the output of AVFS to a different folder than c:\Volumes ?

if so, how would I do that ?

these video files get quite large and I don't have enough space on the c: drive...

TIA.

[joe48]

The c:\Volumes folder is not configurable, but as HolyWu said, no space on the C: drive is actually used regardless of how large the virtual AVI files appear to be.

[Iron_Mike]

The c:\Volumes folder is not configurable, but as HolyWu said, no space on the C: drive is actually used regardless of how large the virtual AVI files appear to be.

thank you, that makes sense ! :)

I went back to an older version of PFM and it allows to mount the file/script within the containing folder...

any concerns using AVFS 1.0.0.6 with an older version of PFM ?

Thanks.

[Myrsloik]

I thought I'd quickly grab the changes for VSFS but pfmmarshaller.h doesn't seem to exist anymore in the build 181 sdk.

Did I miss something or could you do a small revision of AVFS source to work with the latest pismo build?

[asarian]

I have to admit I still don't quite get the point of it all.

I mean, why would I want to open a virtual AVI? So as to parse it to x264? I can do that directly! (Or via avs2yuv pipe). Or so I can load the AVI in a video editor? Why would I then not just create a real output file first?!

Honestly, I don't mean to be negative. In fact, I love the idea! Just not entirely sure how I could use it in a way that would benefit me over what I currently do. Outputting via a virtual file system to a 64-bit version of x264, that I could see happen. :)

Anyways, seems this thread is not seeing all that much activity as it is, so my questions may be moot to start with.

[TheFluff]

it's for tempfile-less interaction with programs which can open avi but do not do so via vfw nor support pipe input

granted, that's a pretty specific use case, but it does solve an existing problem. if you don't have that problem, well, then you don't need this particular bit of software.

[Iron_Mike]

Hi,

I have a question:

does AS always fake output or actually convert the input file by default (w/o specifying specific output parameters) ?

example: When I input a 10-bit RGB (444) file, MediaInfo shows the virtual output file as 8-bit... so is that just the default fake file format done by AS or is the file actually converted to 8-bit ?

Thanks.

[poisondeathray]

Hi,

I have a question:

does AS always fake output or actually convert the input file by default (w/o specifying specific output parameters) ?

example: When I input a 10-bit RGB (444) file, MediaInfo shows the virtual output file as 8-bit... so is that just the default fake file format done by AS or is the file actually converted to 8-bit ?

Thanks.


Avisynth is 8bit natively, it's probably converted by your source plugin

There are "tricks" to load higher bit depth into avisynth (such as stacked 16), but I don't know if they will output a standard format compatible with AVFS, and certainly the receiving program won't understand those hacks

vapoursynth supports higher bit depths natively, it has an VSFS based on AVFS, but I don't think you can get 10bit RGB to work (at least I couldn't, I could get it to load as 10bit RGB in vapoursynth, but not through VSFS) . It might be because I'm running older vpy versions and plugins, not sure

[asarian]

Just to be clear, AVFS will not trash my SSD, right?! (I just don't understand how you can have like a 4G virtual file, without major disk I/O involved).

[TheFluff]

It isn't a 4GB file. That's what "virtual" means - the file doesn't exist. AVSFS files work exactly like any other Avisynth script, the only thing that's different is the interface. There is never anything written to disk.

[Iron_Mike]

Avisynth is 8bit natively, it's probably converted by your source plugin

There are "tricks" to load higher bit depth into avisynth (such as stacked 16), but I don't know if they will output a standard format compatible with AVFS, and certainly the receiving program won't understand those hacks

vapoursynth supports higher bit depths natively, it has an VSFS based on AVFS, but I don't think you can get 10bit RGB to work (at least I couldn't, I could get it to load as 10bit RGB in vapoursynth, but not through VSFS) . It might be because I'm running older vpy versions and plugins, not sure

okay, thanks for explaining that.

here's a test I did:

I fed a 10-bit RGB file to AS and modified the 4CC to fake it as a 10-bit 422 file... which it successfully did - MediaInfo shows it as a 10-bit 422 file.

in this example, what is the actual AS output:

(a) an 8-bit file (downconverted from 10-bit RGB) that AS fakes to be this 10-bit 422 file

(b) it is really the original 10-bit RGB file faked as 10-bit 422

(c) or is it a true 10-bit 422 file, downconverted from 10-bit RGB

?

I somewhat lean towards (a) given your explanation, can you clarify ?

Thanks.

[asarian]

It isn't a 4GB file. That's what "virtual" means - the file doesn't exist. AVSFS files work exactly like any other Avisynth script, the only thing that's different is the interface. There is never anything written to disk.

Just installed it. Didn't use complex things yet, but I have to say, it worked flawlessly! :) Pretty cute like that! Seeking went very well too (this will likely change when I do things like MCTemporalDenoise, or QTGMC, though). Very nice! Thx all!

[poisondeathray]

okay, thanks for explaining that.

here's a test I did:

I fed a 10-bit RGB file to AS and modified the 4CC to fake it as a 10-bit 422 file... which it successfully did - MediaInfo shows it as a 10-bit 422 file.

in this example, what is the actual AS output:

(a) an 8-bit file (downconverted from 10-bit RGB) that AS fakes to be this 10-bit 422 file

(b) it is really the original 10-bit RGB file faked as 10-bit 422

(c) or is it a true 10-bit 422 file, downconverted from 10-bit RGB

?

I somewhat lean towards (a) given your explanation, can you clarify ?

Thanks.


Probably 8bit because it's loaded as 8bit

Normally what people do is use test videos (e.g. 10bit gradients and patterns) , where it's easy to see if the values are valid in the receiving application

10bit 422 YUV source (i.e. "v210") can be done with the VSFS variant, but it's fourcc becomes "P210" (the planar 10bit 422 variant) when fed through VSFS, and most receiving programs don't understand that. How did you change the fourcc ? Did you mean with a hex editor in the source file ?

[kolak]

I don't think you can get 10bit with vsfs using avisynth, as avisythn doesn't natively support any 10bit pixel format on output (I may be wrong here). Other than this I don't think vsfs is written to support any 10bit pixel formats with avisynth.

Use vapoursynth and there you can get few 10bit pixels formats, including mostly supported v210, which Myrsloik kindly added after I asked him.

[poisondeathray]

Use vapoursynth and there you can get few 10bit pixels formats, including mostly supported v210, which Myrsloik kindly added after I asked him.

Were you able to get v210 out of VSFS (I mean through .vpy , not .avs) , vspipe works fine, but it's only "p210" for me in the VSFS virtual AVI

Have you tested 10bit RGB "r210", "r10k" in VSFS ? It loads fine internally as 10bit RGB, but again doesn't work with the VSFS AVI

Or is there a special switch I'm missing ?

(Maybe should post in the vapoursynth section)

[kolak]

v210 works with vpy for sure (just needs special line in the script to enable it). I don't think r210 and r10k work, as these are packed formats, so support for them has to be "added".
In the same time RGB48 and B64A also work if I remember well.

[poisondeathray]

v210 works with vpy for sure (just needs special line in the script to enable it). I don't think r210 and r10k work, as these are packed formats, so support for them has to be "added".
In the same time RGB48 and B64A also work if I remember well.

Yes, they all work in the script (RGB30 for 10bit RGB) , and vspipe out to something like ffmpeg, just not the virtual AVI

I'm asking specifically about VSFS and the virtual AVI, I couldn't even get v210 to work (it "becomes" P210) , even with the v210 switch "enable_v210 = True"

[kolak]

some_clip.set_output()
enable_v210 = True

v210 definitely worked for me (in older vs versions). This has been debugged and tested with many software. Fake v210 AVI worked in Vdub, AE, Premiere, Edius etc. For some software, like Resolve you may need to open fake AVI in QT 7 PRO and save QT reference file.

Maybe Myrsloik broke it :)

[asarian]

I have to admit this is all pretty cute. :) Especially the free seeking is awesome! Not sure how that works, but even while the avs is doing its thing, I can still randomly seek anywhere (like to precisely a troublesome spot). No more carefully select an exact fragment to encode first!

So, thank you for making this!

[asarian]

For users of prior version of AVFS, things have changed. AVFS is now an executable, not a DLL. The installation is now simpler, but some explorer integration is lost.

I'm kinda sad about this. VapourSynth still uses the menu context integration (wth dll), and I find it to be a superior way of mounting things. Is there any way we can get the dll method back?!

[Noctis001]

Hello, I'm trying to mount two or more avs scripts files with avfs in order to work with multiple video files at the same time in Premiere. Do you know a way to do it?
Thank you.

[ultim]

Hello, I'm trying to mount two or more avs scripts files with avfs in order to work with multiple video files at the same time in Premiere. Do you know a way to do it?
Thank you.

Alternatively to avfs, frameserving avs scripts to other applications can also work using the frameserving capabilities of virtualdub.

[poisondeathray]

You can mount multiple scripts at the same time (at least you can with the older version) ; each one will appear in the c:\volumes folder in separate sub folders . Is that what you are asking ? or are you asking how to make them appear in the same directory level ?

[Myrsloik]

I decided to make a small update. The plan is to merge the avfs and vsfs codebases in the end. But first I plan to change the whole thing into something resembling sane C++11 code.

Changes:
Added output support for all 2.6 formats, obviously it no longer works with older versions
Fixed some 64 bit warnings
Fixed headers to match the names modern compilers expect
Simplified a pointlessly complex function

Here's the link (https://dl.dropboxusercontent.com/u/73468194/avfs-test1.7z).

VS2015 runtime required!!!!!!!!!!!!!!!!!!

[Myrsloik]

I've now produced something that you could call a true successor if you consider features. Maybe I'm the new maintainer now or something. I guess.

The source changes can be found in the VapourSynth git (https://github.com/vapoursynth/vapoursynth/tree/master/src/avfs). The same binary supports both avs and vs at once. Doesn't require both to be installed either. Any file ending with .vpy will be treated as a vs script, everything else will be fed to avisynth.

Binaries here (https://dl.dropboxusercontent.com/u/73468194/avfs1.7z). Report bugs or success to help development and so on...

Changes:
Added VaporSynth support with all sane raw fourccs supported
Added output support for all 2.6 formats, obviously it no longer works with older versions
Fixed some 64 bit warnings
Compilable with clang-cl
Fixed headers to match the names modern compilers expect
Simplified a pointlessly complex function

[Myrsloik]

If anyone needs the most recent binaries simply grab the VapourSynth portable packages and extract avfs.exe from there. Needs VS2015 runtime installed.

[tyee]

Just trying this again in a long while. I mount the avs and see the shortcut to the .avs file in the c:\volumes folder but that is all that is there. No avi or wav file. What am I missing?

[Myrsloik]

Just trying this again in a long while. I mount the avs and see the shortcut to the .avs file in the c:\volumes folder but that is all that is there. No avi or wav file. What am I missing?

Look at the text file in the folder to see the error message.

[osgZach]

Myrsloik your version keeps crashing when I try to load a script. Not sure how to get any information out of it so I can give to you.

(I just get the "program has stopped working" modal)

Windows 10 x64
avisynth 32bit.

[Myrsloik]

Myrsloik your version keeps crashing when I try to load a script. Not sure how to get any information out of it so I can give to you.

(I just get the "program has stopped working" modal)

Windows 10 x64
avisynth 32bit.

Start by revealing the exact versions of everything.

[mp3dom]

Same here, avfs 1.0.0.7 (the one provided in the VapourSynth package) and vanilla AviSynth 2.60 x86 on WinServer 2008-R2 x64.
The avs is just a simple call of AVISource to an UTVideo avi (ULY2, so YUY2).

Reverting to 1.0.0.6 and everything returns to work as expected.

[osgZach]

Sorry, yes

OS: Windows 10 x64 Professional
Avisynth: 2.60 Official x32
AVFS: 1.0.0.7 from the R36 Vapoursynth x32 portable package

Same Avisynth trial as the poster above me. only with YV12
I have not tried an older version yet. But standard old old build from AVFS site works for me.
-----------

I need to verify a few things but I suspect the problem may be UTCodec version 17.3.0
I just reverted to 17.2.0 because YV12 was being decoded as YV16, and now its loading a script without crashing.

[Myrsloik]

...

I'll investigate. I think I only really tested with Avs+...

[mp3dom]

I just reverted to 17.2.0 because YV12 was being decoded as YV16, and now its loading a script without crashing.

Uhm, I don't think it's due to UTVideo.
It just crashes even when I try to mount an avs with Colorbars.

This is the error I got (if it may be helpful)

Problem signature:
Problem event name: APPCRASH
Application Name: AVFS.exe
Application Version: 1.0.0.7
Application Timestamp: 5883d60f
Fault Module Name: kernel32.dll
Fault Module Version: 6.1.7601.23572
Fault Module Timestamp: 57fd0378
Exception code: c0000005
Exception Offset: 0001136d
OS Version: 6.1.7601.2.1.0.272.7
Locale ID: 1040
Additional Information 1: 0a9e
Additional Information 2: 0a9e372d3b4ad19135b953a78882e789
Additional Information 3: 0a9e
Additional Information 4: 0a9e372d3b4ad19135b953a78882e789