Avisynth virtual file system

[joe48]

Update
-------
AVFS 1.0.0.6 released, 2015.03.25

The latest release of Avisynth Virtual File System is available at:

http://www.turtlewar.org/avfs/

The package includes a readme.txt file with installation instructions and release notes.

For users of prior version of AVFS, things have changed. AVFS is now an executable, not a DLL. The installation is now simpler, but some explorer integration is lost. Also, the required Pismo File Mount (PFM) build 174 now implements mounting differently, through a c:\volumes folder instead of a virtual mount point placed over the AVS script.

End Update
-----------



I would like to help build an open source or freeware Avisynth virtual file system. This would allow a user to mount an Avisynth script, exposing a virtual uncompressed AVI file through the file system. This would provide alternative solutions for:
- chaining non VFWAPI enabled video processing tools.
- chaining tools that are running on different systems.
- chaining between 32 bit and 64 bit tools.

I have the Windows system extension work completed for this, see http://www.pismotechnic.com/ . If someone is willing to help who has Avisynth/VFWAPI/AVI code experience and resources, I believe things would come together pretty quickly.

[Gerard V]

FWIW this is an excellent concept - provides a means for a Linux port by mounting an avs script as a file system which can be accessed by other apps, bypassing the VFW issues entirely.

[IanB]

This sounds like an automagic MakeAvis implementation, could be very interesting.

This could be a very good way of mapping audio only scripts to a *.WAV file. This has always been a pain as currently they end up appearing as an Audio only *.AVI file, which of course most apps turn there noses up at.

Question :- if you AVFS *.avs files thru to there resulting AV streams how do you propose to edit the underlying script text?

[joe48]

I'm not familiar with AVFS.

With the Pismo File Mount system extension on Windows, files are mounted/unmounted manually by the user, not automatically based on name or type. This is done through mount/unmount commands in explorer context menus, or from the command line through the pfm.exe command. While mounted, the file appears as a folder, the contents of the folder provided by a user mode file system.

A user would edit the .avs file, mount it, access the resulting .avi (or .wav), unmount, edit the .avs file some more, etc..

[IanB]

AVFS="Avisynth virtual file system"

While mounted, the file appears as a folder, the contents of the folder provided by a user mode file system.So you "mount" fred.avs, then access fred.avs\fred.avi or fred.avs\fred.wav

Would it then be possible to access fred.avs\fred.avs so you could edit the script without having to unmount/remount fred.avs (assuming you "compile" the script at each open of fred.avs\fred.avi).

All up this sounds pretty easy. The application opens the fred.avs\fred.avi, internally you have a "map" of the file's .avi structure. As the application read bytes from the file, you PClip->GetFrame() and/or PClip->GetAudio() to populate the user buffer with the appropriate data.

What do you need? Is the Pismo source available?

[joe48]

Ian,

I did a quick search for AVFS after your first post. It hit on a Linux project that allows transparent mount of archive file contents. I guess that wasn't what you were referring to. I like the acronym AVFS. It's not like it is really possible to avoid acronym collisions.

I think we're on the same page. The file system could be written a number of different ways. Like you mention, the source for the avs script could be echoed into a virtual file to allow it to be edited without unmounting. Could also dynamically change portions of the script based on the name of the output file that is opened, so opening the out.avi would run a slightly different script than opening out.wav.

fred.avs
+- script.avs
+- out.avi
+- out.wav
(or fred.avi, fred.wav, like your example)

To get started, a basic mode would be a single output avi file with Avisynth initialized only at the initial mount. This would avoid multiple instances of the filters running in a single process. Maybe that's not an issue.

[IanB]

So where to from here?

[joe48]

What I need to get started is another developer who can do the things you mentioned earlier.

Most of what I'll provide is already included in the Pismo File Mount Audit Package. This package includes a developer document, headers files, and a few C/C++ sample formatters (user mode file systems). Still, I can speed up the PFM specific parts of the development, do troubleshooting, etc..

To start with I can put together the basic AVFS formatter source, minus the actual AVI/Avisynth bits. I need to know the general parameters first:

C or C++?
Allow edit of AVS file while mounted?
Virtual file naming conventions?
Source code repository?
AVS file type recognition strategy?
Tools? (I prefer the SDK compilers myself)

The Pismo File Mount Audit Package can be downloaded from http://www.pismotechnic.com/download/ . It is a freeware utility that allows mounting if various archive file formats, and includes the PFM development kit. The audit package exists and is licensed for free use by individuals, business users, developers, and to be used as an enabling system extension for other freeware and open source projects.

If you are willing to work with me on this, send me a PM and we can continue offline.

[hanfrunz]

i came up with similar ideas years ago, hopefully now someone is coding it :-) here are the links to older threads:

http://forum.doom9.org/showthread.php?t=44028
http://forum.doom9.org/showthread.php?t=49383
http://forum.doom9.org/showthread.php?t=103426


i'm not a filessystem/driver programmer, but i'll help whereever i can.

regards,
hanfrunz

[joe48]

Coding is in progress. The driver bits were already done. AVFS is pretty simple as file systems go. Things should come together pretty quickly.

I expect some longer term work will be needed to add output file format options, to allow compatibility with more video processing programs.

[sh0dan]

Feel free to contact me, if you need additional, or help coding some of it.

I can help with various audio formats, and avisynth internals.

[Fizick]

Probably it was related discusion?
http://forum.doom9.org/showthread.php?t=108653

[joe48]

I hadn't read that thread. It is related.


I have finished the file system bits for AVFS. The code is on sourceforge in the module avfs at the root of the avisynth2 CVS repository. It builds with MSVC6 and has been through some developer testing.

http://avisynth2.cvs.sourceforge.net/avisynth2/avfs/

What is finished is a virtual file system that allows you to "mount" an avs script, turning it into a folder that contains an empty virtual media file and an editable copy of the script file.

Not finished:
1. Determine the initial names and sizes of the virtual media file(s).
- use stream info from avisynth and options from avs file to determine appropriate file format for virtual media file.
- support spanned media files, for the varios 32 bit file size limitations.
- support different media file formats, wav, avi1, avi2.
2. Handle virtual file read requests.
- figure out which frames and audio are needed for a range of bytes in the virtual file.
- read frames and audio from avisynth and fill the read buffer.
- fabricate any file or frame headers needed for portions of the read buffer.

Anyone interested in working on these remaining bits, send me a PM.

[hanfrunz]

two wishes:

-support for uncompressed bitmap sequences (TIFF, TGA, YUV, ...)
-support for uncompressed Quicktime-MOV

regards
hanfrunz

[IanB]

@Hanfrunz,

It would help if you did the google homework and find the references needed to write the file formats.

[Zarxrax]

Hmmm, would it be possible for this to support multiple files?
For instance, if I have a folder of 100 avisynth scripts, could they all be mounted?

[joe48]

Mounting and unmounting files is a manual action, but you can simultaneously mount as many files as you want. Each mount is handled by an instance of AVFS+Avisynth running in a dedicated pfmhost.exe process. You will be able to do things like feed the output of 2 scripts into a 3rd script.

[Mug Funky]

on quicktime support:

http://forum.doom9.org/showthread.php?t=104293

there should be some useful stuff there about stuffing yuy2 into a mov file, although i think it uses quicktime itself to handle it, so it may not work for virtual filesystem stuff.

one question:

would this work over a network? could i for example create an AVFS folder on the SAN here from one computer (an 8-core PC with dual fibre-channel connection to the SAN), and load it on another machine (a mac edit suite)?

it'd be truly awesome to be able to dump an avs straight into final cut pro, and render it there for realtime playback, especially considering the colour problems with quicktime (ie, if you use quicktime to export a mov, it'll bugger up the colours in a peculiar and unrecoverable way - something like YUV to RGB with made-up coefficients and some kind of gamma "correction").

i'm not sure if this would be possible though, bearing in mind the network share in question is a mac one, running who-knows-what filesystem.

[IanB]

@Mug Funky,

Ta! muchly (but the idea was to make Hanfrunz earn his keep :D )

Well we have the .WAV file virtualization working (this is a trivial file format) and as far as I can see W2K3 (at least) cannot tell there is anything different from real files in the same file tree. In fact for testing as this box has no sound card I shared the upper directory and accessed the virtual files remotely. Beeeuuudy!

[squid_80]

Sorry to be a spoilsport, but what happens if the .wav file is >4GB (5.1 channels etc.)?

[hanfrunz]

hello,

here are some fileformat specs:

TGA (http://www.dca.fee.unicamp.br/~martino/disciplinas/ea978/tgaffs.pdf)
TIFF (http://partners.adobe.com/public/developer/tiff/index.html)

regards,
hanfrunz

[Shinigami-Sama]

just to be a stick in the mud
will the eventually be made cross platform, or would that be to much effort?

[Adub]

I sure hope so. I am getting more and more excited about this, especially if this were to be used in conjunction with David Horman's Avisynth cluster plugin. He said he was having problems with networking file access. This should make things a little bit easier.

[joe48]

Network access of AVFS files will work fine so long as the user creating the mount has administrator privileges, and the mount is created with "everyone read" access.

A Linux version of AVFS is feasible, assuming a native Linux port of Avisynth. The PFM/Windows specific code in AVFS would be replaced with Linux/Fuse code.

[IanB]

@Squid,

Current design provides sequential segmented wave files when total size exceeds 2GB. Longer term we may provide alternate formats like .W64 that is popular and doesn't have a limit.

1st cut we will probably provide a single .WAV for 0 to 4Gb and 1GB? segments for 1GB-> hugh (it costs nothing to provide parallel virtualizations).

In the 1st cut of .AVI will follow the same size modeling. Doing >4gb .AVI files involves an extra indexing and internal segmentation layer, that will make a quick implementation into an epic saga. So hugh .AVI will be phase 2.

[IanB]

@hanfrunz,

But you said you wanted .MOV :D

[hanfrunz]

@hanfrunz,

But you said you wanted .MOV :D

of course:
Official Apple Developer Site (http://developer.apple.com/documentation/QuickTime/)
Quicktime Fileformat (pdf) (http://developer.apple.com/documentation/QuickTime/QTFF/qtff.pdf)

:)

[Mug Funky]

i'm really looking forward to seeing some kind of quicktime support :)

not to hassle anybody of course, but given the current state of quicktime, avisynth + qtinput/qtoutput is the only way to avoid colour shifts within quicktime. this is a massive problem that crApple steadfastly refuse to even admit the existence of, but can be plainly seen even on uncalibrated monitors with an untrained eye.

...hmm, seems i'm offloading some of my frustration to the wrong place :).

basically, an AVFS would give a working link between the grading suite and the online suite without having to go to a tape intermediate (which has it's own problems, thankyou very much nVidia/Assimilate). it would also be quite fast - just drop the virtual mov in, render, then output. rather than hours of fannying about with image sequences in final cu't pro.

[Gerard V]

JMTCW. To keep this practical and to avoid feature creep, I would recommend that the initial project limit itself to whatever capabilities a chosen version of Avisynth has built in, and works to deliver that via a VFS on Linux. Call that phase 1. When that works, additonal capabilities can be added, and ports to other OSes envisgaed. Too much too soon will kill this project which has somewhere between 0.5 and 2 developers based on my assessment of the posts above. Keep it simple or it'll remain in the just another good idea state.

[hanfrunz]

Uuhm is there any progress with this tool? I love the idea of a virtual FS and it could be so usefull in a production enviroment. Think of serving a DVD perfectly resized and processed to a discreet inferno 2k project via ftp (TGA-Sequence) with a normal Workstation without TB of harddiskspace...

regards
hanfrunz

[joe48]

The first release of Avisynth Virtual File System is now available. The package includes a readme.txt file with brief installation instructions. It is available for the time being at:

http://www.turtlewar.org/avfs/

Remember, this is a first release. Chew slowly.

Getting it to this stage has taken a significant amount of work by Ian B and myself.

Joe L.

[Shinigami-Sama]

question
can this be chained?

as in
PC1 -> de-interlace -> resize -> share
PC2 -> mounts share -> processes ( sharpen/denoise ect)

if so I might have to start collecting those relativly low powered PCs again

[joe48]

question
can this be chained?

as in
PC1 -> de-interlace -> resize -> share
PC2 -> mounts share -> processes ( sharpen/denoise ect)


I hope so. Someone still needs to prove it. I have fed a virtual AVI back into a second script, and a few months back I tested through a file share.

Joe L.

[Shinigami-Sama]

I hope so. Someone still needs to prove it. I have fed a virtual AVI back into a second script, and a few months back I tested through a file share.

Joe L.

awesome
may be able to put some older hardware to good use again with this

[hanfrunz]

:D

great!!!

i hope i understand the source and can help with further file formats.
Could you write a simple sample for let's say an uncompressed raw RGB32 File, so new developers can quickly understand how all this works?
And is it possible to generate image sequences in an extra folder?

EDIT:
using this very simple script: colorbars(320,240)
i get an error with virtualdub 1.7.8. (stable): AVI Truncated or invalid audio format (16 byte, should be 18).
Fileinfo says Compression: Unknown (tag 0003)


regards
hanfrunz

[joe48]

And is it possible to generate image sequences in an extra folder?


Changes are needed to support virtual files in a sub-folder. I might work on this next, whenever next is.


using this very simple script: colorbars(320,240)
i get an error with virtualdub 1.7.8. (stable): AVI Truncated or invalid audio format (16 byte, should be 18).


I missed adding a sample.avs to the package. Try this:
--- clip ---
ColorBars()
ConvertAudioTo16bit()
ConvertToRGB24()
--- clip ---

Joe L.

[Zarxrax]

Wow, this is awesome! I have a few questions:
I notice it creates some fake uncompressed files. Do these files actually take up the amount of space on your hard disk that they report?
Does this create any performance hit compared to accessing the avs directly?
Will there be any problems mounting and accessing many avs scripts at the same time with this?

[joe48]

Do these files actually take up the amount of space on your hard disk that they report?

Data comes straight from Avisynth, no intermediate hard disk space is used.

Does this create any performance hit compared to accessing the avs directly?

Ian did some perf tests a while back, indicated it was slightly slower. If anyone runs some perf tests they should post numbers.

Will there be any problems mounting and accessing many avs scripts at the same time with this?

No arbitrary limits on number of mounted scripts. Each script is handled by a separate process.

[pwolfamv]

This is awesome, great work. I did a few tests. The one i'm most concerned about is an HD source test which is mostly about loading time.

My source is 1920x1080 and the filesize in the vfs is about 400gb. It took about 5-10 minutes to load it into vdub compared to the seconds it takes to load just the script (or a file of similar size). Of course, after loading the file, it seeked like normal.

Is this something that can be worked on, or is this just a limitation of the virtual file system, that anyone who uses this will have to deal with?

[joe48]

It took about 5-10 minutes to load it into vdub compared to the seconds it takes to load just the script (or a file of similar size).

Script loading should not be any slower with AVFS. I am seeing script loading take ~5 sec per DirectShowSource() in the script, with similar load times for AVFS as for reading the script directly with Virtualdub.

[pwolfamv]

Script loading should not be any slower with AVFS. I am seeing script loading take ~5 sec per DirectShowSource() in the script, with similar load times for AVFS as for reading the script directly with Virtualdub.

I'm just letting you know. The test i did was using a 90gb lagarith encoded avi (1920x1080) using avisource().

I did another test with the original EVO sources using ffmpegsource and it took about a minute or two to load. the file size in the vfs was about half the size.

[Mug Funky]

haha! sweet. now i can load any video i want into scratch via quicktime and converttorgb24(). excellent.

[edit]

even better, if i share this on the network it still works! i can load avs files into final cut pro and play them to digi now, so long as the processing and network speed are fast enough (which should be fine for standard def at least).

in case anyone wants to do this:

- make your avs, put "converttoyuy2()" at the end. make sure the avs file is somewhere shared on the network.
- mount it the long way, so you can give "everyone" read permission
- go to your mac, open the avi file, ignore the "baaw! no codec" error, save a reference movie, open that ref movie in textedit, searchreplace "YUY2" with "yuvs" (this is case sensitive), and it should magically work. with FCP in "unlimited RT" you may be able to play out as-is, otherwise just render it and know it's more reliable than qtoutput...

i haven't tried ingesting into avid yet, but i'm sure that works too.

[Smetvid]

This thing works great so far!

I tried it with Avid Liquid by treating the clip as RGB. Avid Liquid never works with anything and it will only load DV, RGB and mjpeg avi files. The clips even come in with the correct properties and settings. The only problem is they are very very slow. I get about 1 fps playback with these clips even though I am just feeding a mjpeg sample that plays perfectly fine in Liquid when I import it directly. The only problem is that Liquid thinks it is a realtime clip and it expects it to playback at full speed. The only way to do a force render is to put an effect on the clip. I never expected this stuff to playback in RT but it would be nice if it could be a little bit faster.

[joe48]

I get about 1 fps playback with these clips even though


Is the 1fps due to processing in the script, or do you think there is some sort of compatibility problem?

Things you could try:
- Try the virtual AVI in VirtualDub, see if it plays faster.
- Copy the virtual AVI to a real file and try it.
- Save the virtual AVI through VirtualDub and try it. VirtualDub has more mature AVI formatting logic.
- Monitor pfmhost.exe (and ...) CPU usage in task manager.

Joe L.

[Zarxrax]

The only problem is that Liquid thinks it is a realtime clip and it expects it to playback at full speed. The only way to do a force render is to put an effect on the clip. I never expected this stuff to playback in RT but it would be nice if it could be a little bit faster.

That sounds exactly like a problem that Premiere Pro has if you are using the avisynth script import plugin with it. It "expects" the clips to be able to play in realtime, but they often can't, and you have to put an effect on them to make it render.

[Smetvid]

To be totally fair this is a 1920x1080 clip and perhaps avisynth just doesn't handle HD all that fast.

I tried the clip in virtual dub and it plays a little bit faster but not much. I think it is around 5 fps.

cpu usage is around 42% on my single 3.0 Ghz cpu with 2 GB of ram.

Again this HD mjpeg clip will play perfectly fine on this system in it's native form.

[Shinigami-Sama]

To be totally fair this is a 1920x1080 clip and perhaps avisynth just doesn't handle HD all that fast.

I tried the clip in virtual dub and it plays a little bit faster but not much. I think it is around 5 fps.

cpu usage is around 42% on my single 3.0 Ghz cpu with 2 GB of ram.

Again this HD mjpeg clip will play perfectly fine on this system in it's native form.
try turning off HT

I had to do that a few times to get certain clips to play smoothly

[joe48]

Available at: http://www.turtlewar.org/avfs/

Changes
--------

- Added background read-ahead cache to improve CPU utilization on multi-core systems or when serving data through the network. Includes some tuneability via AVFS_ReadAheadFrameCount script variable.

- Added AVFS_AVI_VidFcc script variable, to allow overriding video format fourcc code. Useful to improve compatibility with specific encoders or readers that do not recognize the YUY2 or YV12 fourcc codes used by default.

- Added some diagnostic/status tracing, visible using Pismo Trace Monitor. Shows the frame number and audio sample offsets that are being accessed to satisfy read requests.

[mikeytown2]

Running andLinux (http://www.andlinux.org/) Beta 1 (04/21/2008), and using cofs (http://colinux.wikia.com/wiki/Cofs_device) to access the NTFS drive, andLinux can not read the fake dir. This is not a show stopper because it can be read using a sambia share. Just thought i would let you know. The problem probably exists in cofs since direct NTFS reading is still experimental.
http://img176.imageshack.us/img176/1474/andlinuxavsfsjo1.th.jpg (http://img176.imageshack.us/my.php?image=andlinuxavsfsjo1.jpg)

Thanks for developing AVFS, its quite amazing!

[joe48]

Running andLinux Beta 1 (04/21/2008), and using cofs to access the NTFS drive, andLinux can not read the fake dir.


I can guess as to why COFS access does not work. The Windows half of the COFS code may open files as system instead of as the user who is running Linux. If so then there will be other effects, like the inability to access NTFS encrypted files, bypassing some 3rd party Windows security software, maybe even privilege escalation vulnerabilities. Anyway, just guessing.

A work-around in Pismo File Mount is probably possible. It is not an AVFS specific problem.