Avisynth virtual file system

[joe48]

Update
-------
AVFS 1.0.0.5 released, 2010.05.23

The latest release of Avisynth Virtual File System is available at:

http://www.turtlewar.org/avfs/

The package includes a readme.txt file with installation instructions and release notes.

End Update
-----------



I would like to help build an open source or freeware Avisynth virtual file system. This would allow a user to mount an Avisynth script, exposing a virtual uncompressed AVI file through the file system. This would provide alternative solutions for:
- chaining non VFWAPI enabled video processing tools.
- chaining tools that are running on different systems.
- chaining between 32 bit and 64 bit tools.

I have the Windows system extension work completed for this, see http://www.pismotechnic.com/ . If someone is willing to help who has Avisynth/VFWAPI/AVI code experience and resources, I believe things would come together pretty quickly.

[Gerard V]

FWIW this is an excellent concept - provides a means for a Linux port by mounting an avs script as a file system which can be accessed by other apps, bypassing the VFW issues entirely.

[IanB]

This sounds like an automagic MakeAvis implementation, could be very interesting.

This could be a very good way of mapping audio only scripts to a *.WAV file. This has always been a pain as currently they end up appearing as an Audio only *.AVI file, which of course most apps turn there noses up at.

Question :- if you AVFS *.avs files thru to there resulting AV streams how do you propose to edit the underlying script text?

[joe48]

I'm not familiar with AVFS.

With the Pismo File Mount system extension on Windows, files are mounted/unmounted manually by the user, not automatically based on name or type. This is done through mount/unmount commands in explorer context menus, or from the command line through the pfm.exe command. While mounted, the file appears as a folder, the contents of the folder provided by a user mode file system.

A user would edit the .avs file, mount it, access the resulting .avi (or .wav), unmount, edit the .avs file some more, etc..

[IanB]

AVFS="Avisynth virtual file system"

While mounted, the file appears as a folder, the contents of the folder provided by a user mode file system.So you "mount" fred.avs, then access fred.avs\fred.avi or fred.avs\fred.wav

Would it then be possible to access fred.avs\fred.avs so you could edit the script without having to unmount/remount fred.avs (assuming you "compile" the script at each open of fred.avs\fred.avi).

All up this sounds pretty easy. The application opens the fred.avs\fred.avi, internally you have a "map" of the file's .avi structure. As the application read bytes from the file, you PClip->GetFrame() and/or PClip->GetAudio() to populate the user buffer with the appropriate data.

What do you need? Is the Pismo source available?

[joe48]

Ian,

I did a quick search for AVFS after your first post. It hit on a Linux project that allows transparent mount of archive file contents. I guess that wasn't what you were referring to. I like the acronym AVFS. It's not like it is really possible to avoid acronym collisions.

I think we're on the same page. The file system could be written a number of different ways. Like you mention, the source for the avs script could be echoed into a virtual file to allow it to be edited without unmounting. Could also dynamically change portions of the script based on the name of the output file that is opened, so opening the out.avi would run a slightly different script than opening out.wav.

fred.avs
+- script.avs
+- out.avi
+- out.wav
(or fred.avi, fred.wav, like your example)

To get started, a basic mode would be a single output avi file with Avisynth initialized only at the initial mount. This would avoid multiple instances of the filters running in a single process. Maybe that's not an issue.

[IanB]

So where to from here?

[joe48]

What I need to get started is another developer who can do the things you mentioned earlier.

Most of what I'll provide is already included in the Pismo File Mount Audit Package. This package includes a developer document, headers files, and a few C/C++ sample formatters (user mode file systems). Still, I can speed up the PFM specific parts of the development, do troubleshooting, etc..

To start with I can put together the basic AVFS formatter source, minus the actual AVI/Avisynth bits. I need to know the general parameters first:

C or C++?
Allow edit of AVS file while mounted?
Virtual file naming conventions?
Source code repository?
AVS file type recognition strategy?
Tools? (I prefer the SDK compilers myself)

The Pismo File Mount Audit Package can be downloaded from http://www.pismotechnic.com/download/ . It is a freeware utility that allows mounting if various archive file formats, and includes the PFM development kit. The audit package exists and is licensed for free use by individuals, business users, developers, and to be used as an enabling system extension for other freeware and open source projects.

If you are willing to work with me on this, send me a PM and we can continue offline.

[hanfrunz]

i came up with similar ideas years ago, hopefully now someone is coding it :-) here are the links to older threads:

http://forum.doom9.org/showthread.php?t=44028
http://forum.doom9.org/showthread.php?t=49383
http://forum.doom9.org/showthread.php?t=103426


i'm not a filessystem/driver programmer, but i'll help whereever i can.

regards,
hanfrunz

[joe48]

Coding is in progress. The driver bits were already done. AVFS is pretty simple as file systems go. Things should come together pretty quickly.

I expect some longer term work will be needed to add output file format options, to allow compatibility with more video processing programs.

[sh0dan]

Feel free to contact me, if you need additional, or help coding some of it.

I can help with various audio formats, and avisynth internals.

[Fizick]

Probably it was related discusion?
http://forum.doom9.org/showthread.php?t=108653

[joe48]

I hadn't read that thread. It is related.


I have finished the file system bits for AVFS. The code is on sourceforge in the module avfs at the root of the avisynth2 CVS repository. It builds with MSVC6 and has been through some developer testing.

http://avisynth2.cvs.sourceforge.net/avisynth2/avfs/

What is finished is a virtual file system that allows you to "mount" an avs script, turning it into a folder that contains an empty virtual media file and an editable copy of the script file.

Not finished:
1. Determine the initial names and sizes of the virtual media file(s).
- use stream info from avisynth and options from avs file to determine appropriate file format for virtual media file.
- support spanned media files, for the varios 32 bit file size limitations.
- support different media file formats, wav, avi1, avi2.
2. Handle virtual file read requests.
- figure out which frames and audio are needed for a range of bytes in the virtual file.
- read frames and audio from avisynth and fill the read buffer.
- fabricate any file or frame headers needed for portions of the read buffer.

Anyone interested in working on these remaining bits, send me a PM.

[hanfrunz]

two wishes:

-support for uncompressed bitmap sequences (TIFF, TGA, YUV, ...)
-support for uncompressed Quicktime-MOV

regards
hanfrunz

[IanB]

@Hanfrunz,

It would help if you did the google homework and find the references needed to write the file formats.

[Zarxrax]

Hmmm, would it be possible for this to support multiple files?
For instance, if I have a folder of 100 avisynth scripts, could they all be mounted?

[joe48]

Mounting and unmounting files is a manual action, but you can simultaneously mount as many files as you want. Each mount is handled by an instance of AVFS+Avisynth running in a dedicated pfmhost.exe process. You will be able to do things like feed the output of 2 scripts into a 3rd script.

[Mug Funky]

on quicktime support:

http://forum.doom9.org/showthread.php?t=104293

there should be some useful stuff there about stuffing yuy2 into a mov file, although i think it uses quicktime itself to handle it, so it may not work for virtual filesystem stuff.

one question:

would this work over a network? could i for example create an AVFS folder on the SAN here from one computer (an 8-core PC with dual fibre-channel connection to the SAN), and load it on another machine (a mac edit suite)?

it'd be truly awesome to be able to dump an avs straight into final cut pro, and render it there for realtime playback, especially considering the colour problems with quicktime (ie, if you use quicktime to export a mov, it'll bugger up the colours in a peculiar and unrecoverable way - something like YUV to RGB with made-up coefficients and some kind of gamma "correction").

i'm not sure if this would be possible though, bearing in mind the network share in question is a mac one, running who-knows-what filesystem.

[IanB]

@Mug Funky,

Ta! muchly (but the idea was to make Hanfrunz earn his keep :D )

Well we have the .WAV file virtualization working (this is a trivial file format) and as far as I can see W2K3 (at least) cannot tell there is anything different from real files in the same file tree. In fact for testing as this box has no sound card I shared the upper directory and accessed the virtual files remotely. Beeeuuudy!

[squid_80]

Sorry to be a spoilsport, but what happens if the .wav file is >4GB (5.1 channels etc.)?

[hanfrunz]

hello,

here are some fileformat specs:

TGA (http://www.dca.fee.unicamp.br/~martino/disciplinas/ea978/tgaffs.pdf)
TIFF (http://partners.adobe.com/public/developer/tiff/index.html)

regards,
hanfrunz

[Shinigami-Sama]

just to be a stick in the mud
will the eventually be made cross platform, or would that be to much effort?

[Adub]

I sure hope so. I am getting more and more excited about this, especially if this were to be used in conjunction with David Horman's Avisynth cluster plugin. He said he was having problems with networking file access. This should make things a little bit easier.

[joe48]

Network access of AVFS files will work fine so long as the user creating the mount has administrator privileges, and the mount is created with "everyone read" access.

A Linux version of AVFS is feasible, assuming a native Linux port of Avisynth. The PFM/Windows specific code in AVFS would be replaced with Linux/Fuse code.

[IanB]

@Squid,

Current design provides sequential segmented wave files when total size exceeds 2GB. Longer term we may provide alternate formats like .W64 that is popular and doesn't have a limit.

1st cut we will probably provide a single .WAV for 0 to 4Gb and 1GB? segments for 1GB-> hugh (it costs nothing to provide parallel virtualizations).

In the 1st cut of .AVI will follow the same size modeling. Doing >4gb .AVI files involves an extra indexing and internal segmentation layer, that will make a quick implementation into an epic saga. So hugh .AVI will be phase 2.

[IanB]

@hanfrunz,

But you said you wanted .MOV :D

[hanfrunz]

@hanfrunz,

But you said you wanted .MOV :D

of course:
Official Apple Developer Site (http://developer.apple.com/documentation/QuickTime/)
Quicktime Fileformat (pdf) (http://developer.apple.com/documentation/QuickTime/QTFF/qtff.pdf)

:)

[Mug Funky]

i'm really looking forward to seeing some kind of quicktime support :)

not to hassle anybody of course, but given the current state of quicktime, avisynth + qtinput/qtoutput is the only way to avoid colour shifts within quicktime. this is a massive problem that crApple steadfastly refuse to even admit the existence of, but can be plainly seen even on uncalibrated monitors with an untrained eye.

...hmm, seems i'm offloading some of my frustration to the wrong place :).

basically, an AVFS would give a working link between the grading suite and the online suite without having to go to a tape intermediate (which has it's own problems, thankyou very much nVidia/Assimilate). it would also be quite fast - just drop the virtual mov in, render, then output. rather than hours of fannying about with image sequences in final cu't pro.

[Gerard V]

JMTCW. To keep this practical and to avoid feature creep, I would recommend that the initial project limit itself to whatever capabilities a chosen version of Avisynth has built in, and works to deliver that via a VFS on Linux. Call that phase 1. When that works, additonal capabilities can be added, and ports to other OSes envisgaed. Too much too soon will kill this project which has somewhere between 0.5 and 2 developers based on my assessment of the posts above. Keep it simple or it'll remain in the just another good idea state.

[hanfrunz]

Uuhm is there any progress with this tool? I love the idea of a virtual FS and it could be so usefull in a production enviroment. Think of serving a DVD perfectly resized and processed to a discreet inferno 2k project via ftp (TGA-Sequence) with a normal Workstation without TB of harddiskspace...

regards
hanfrunz

[joe48]

The first release of Avisynth Virtual File System is now available. The package includes a readme.txt file with brief installation instructions. It is available for the time being at:

http://www.turtlewar.org/avfs/

Remember, this is a first release. Chew slowly.

Getting it to this stage has taken a significant amount of work by Ian B and myself.

Joe L.

[Shinigami-Sama]

question
can this be chained?

as in
PC1 -> de-interlace -> resize -> share
PC2 -> mounts share -> processes ( sharpen/denoise ect)

if so I might have to start collecting those relativly low powered PCs again

[joe48]

question
can this be chained?

as in
PC1 -> de-interlace -> resize -> share
PC2 -> mounts share -> processes ( sharpen/denoise ect)


I hope so. Someone still needs to prove it. I have fed a virtual AVI back into a second script, and a few months back I tested through a file share.

Joe L.

[Shinigami-Sama]

I hope so. Someone still needs to prove it. I have fed a virtual AVI back into a second script, and a few months back I tested through a file share.

Joe L.

awesome
may be able to put some older hardware to good use again with this

[hanfrunz]

:D

great!!!

i hope i understand the source and can help with further file formats.
Could you write a simple sample for let's say an uncompressed raw RGB32 File, so new developers can quickly understand how all this works?
And is it possible to generate image sequences in an extra folder?

EDIT:
using this very simple script: colorbars(320,240)
i get an error with virtualdub 1.7.8. (stable): AVI Truncated or invalid audio format (16 byte, should be 18).
Fileinfo says Compression: Unknown (tag 0003)


regards
hanfrunz

[joe48]

And is it possible to generate image sequences in an extra folder?


Changes are needed to support virtual files in a sub-folder. I might work on this next, whenever next is.


using this very simple script: colorbars(320,240)
i get an error with virtualdub 1.7.8. (stable): AVI Truncated or invalid audio format (16 byte, should be 18).


I missed adding a sample.avs to the package. Try this:
--- clip ---
ColorBars()
ConvertAudioTo16bit()
ConvertToRGB24()
--- clip ---

Joe L.

[Zarxrax]

Wow, this is awesome! I have a few questions:
I notice it creates some fake uncompressed files. Do these files actually take up the amount of space on your hard disk that they report?
Does this create any performance hit compared to accessing the avs directly?
Will there be any problems mounting and accessing many avs scripts at the same time with this?

[joe48]

Do these files actually take up the amount of space on your hard disk that they report?

Data comes straight from Avisynth, no intermediate hard disk space is used.

Does this create any performance hit compared to accessing the avs directly?

Ian did some perf tests a while back, indicated it was slightly slower. If anyone runs some perf tests they should post numbers.

Will there be any problems mounting and accessing many avs scripts at the same time with this?

No arbitrary limits on number of mounted scripts. Each script is handled by a separate process.

[pwolfamv]

This is awesome, great work. I did a few tests. The one i'm most concerned about is an HD source test which is mostly about loading time.

My source is 1920x1080 and the filesize in the vfs is about 400gb. It took about 5-10 minutes to load it into vdub compared to the seconds it takes to load just the script (or a file of similar size). Of course, after loading the file, it seeked like normal.

Is this something that can be worked on, or is this just a limitation of the virtual file system, that anyone who uses this will have to deal with?

[joe48]

It took about 5-10 minutes to load it into vdub compared to the seconds it takes to load just the script (or a file of similar size).

Script loading should not be any slower with AVFS. I am seeing script loading take ~5 sec per DirectShowSource() in the script, with similar load times for AVFS as for reading the script directly with Virtualdub.

[pwolfamv]

Script loading should not be any slower with AVFS. I am seeing script loading take ~5 sec per DirectShowSource() in the script, with similar load times for AVFS as for reading the script directly with Virtualdub.

I'm just letting you know. The test i did was using a 90gb lagarith encoded avi (1920x1080) using avisource().

I did another test with the original EVO sources using ffmpegsource and it took about a minute or two to load. the file size in the vfs was about half the size.

[Mug Funky]

haha! sweet. now i can load any video i want into scratch via quicktime and converttorgb24(). excellent.

[edit]

even better, if i share this on the network it still works! i can load avs files into final cut pro and play them to digi now, so long as the processing and network speed are fast enough (which should be fine for standard def at least).

in case anyone wants to do this:

- make your avs, put "converttoyuy2()" at the end. make sure the avs file is somewhere shared on the network.
- mount it the long way, so you can give "everyone" read permission
- go to your mac, open the avi file, ignore the "baaw! no codec" error, save a reference movie, open that ref movie in textedit, searchreplace "YUY2" with "yuvs" (this is case sensitive), and it should magically work. with FCP in "unlimited RT" you may be able to play out as-is, otherwise just render it and know it's more reliable than qtoutput...

i haven't tried ingesting into avid yet, but i'm sure that works too.

[Smetvid]

This thing works great so far!

I tried it with Avid Liquid by treating the clip as RGB. Avid Liquid never works with anything and it will only load DV, RGB and mjpeg avi files. The clips even come in with the correct properties and settings. The only problem is they are very very slow. I get about 1 fps playback with these clips even though I am just feeding a mjpeg sample that plays perfectly fine in Liquid when I import it directly. The only problem is that Liquid thinks it is a realtime clip and it expects it to playback at full speed. The only way to do a force render is to put an effect on the clip. I never expected this stuff to playback in RT but it would be nice if it could be a little bit faster.

[joe48]

I get about 1 fps playback with these clips even though


Is the 1fps due to processing in the script, or do you think there is some sort of compatibility problem?

Things you could try:
- Try the virtual AVI in VirtualDub, see if it plays faster.
- Copy the virtual AVI to a real file and try it.
- Save the virtual AVI through VirtualDub and try it. VirtualDub has more mature AVI formatting logic.
- Monitor pfmhost.exe (and ...) CPU usage in task manager.

Joe L.

[Zarxrax]

The only problem is that Liquid thinks it is a realtime clip and it expects it to playback at full speed. The only way to do a force render is to put an effect on the clip. I never expected this stuff to playback in RT but it would be nice if it could be a little bit faster.

That sounds exactly like a problem that Premiere Pro has if you are using the avisynth script import plugin with it. It "expects" the clips to be able to play in realtime, but they often can't, and you have to put an effect on them to make it render.

[Smetvid]

To be totally fair this is a 1920x1080 clip and perhaps avisynth just doesn't handle HD all that fast.

I tried the clip in virtual dub and it plays a little bit faster but not much. I think it is around 5 fps.

cpu usage is around 42% on my single 3.0 Ghz cpu with 2 GB of ram.

Again this HD mjpeg clip will play perfectly fine on this system in it's native form.

[Shinigami-Sama]

To be totally fair this is a 1920x1080 clip and perhaps avisynth just doesn't handle HD all that fast.

I tried the clip in virtual dub and it plays a little bit faster but not much. I think it is around 5 fps.

cpu usage is around 42% on my single 3.0 Ghz cpu with 2 GB of ram.

Again this HD mjpeg clip will play perfectly fine on this system in it's native form.
try turning off HT

I had to do that a few times to get certain clips to play smoothly

[joe48]

Available at: http://www.turtlewar.org/avfs/

Changes
--------

- Added background read-ahead cache to improve CPU utilization on multi-core systems or when serving data through the network. Includes some tuneability via AVFS_ReadAheadFrameCount script variable.

- Added AVFS_AVI_VidFcc script variable, to allow overriding video format fourcc code. Useful to improve compatibility with specific encoders or readers that do not recognize the YUY2 or YV12 fourcc codes used by default.

- Added some diagnostic/status tracing, visible using Pismo Trace Monitor. Shows the frame number and audio sample offsets that are being accessed to satisfy read requests.

[mikeytown2]

Running andLinux (http://www.andlinux.org/) Beta 1 (04/21/2008), and using cofs (http://colinux.wikia.com/wiki/Cofs_device) to access the NTFS drive, andLinux can not read the fake dir. This is not a show stopper because it can be read using a sambia share. Just thought i would let you know. The problem probably exists in cofs since direct NTFS reading is still experimental.
http://img176.imageshack.us/img176/1474/andlinuxavsfsjo1.th.jpg (http://img176.imageshack.us/my.php?image=andlinuxavsfsjo1.jpg)

Thanks for developing AVFS, its quite amazing!

[joe48]

Running andLinux Beta 1 (04/21/2008), and using cofs to access the NTFS drive, andLinux can not read the fake dir.


I can guess as to why COFS access does not work. The Windows half of the COFS code may open files as system instead of as the user who is running Linux. If so then there will be other effects, like the inability to access NTFS encrypted files, bypassing some 3rd party Windows security software, maybe even privilege escalation vulnerabilities. Anyway, just guessing.

A work-around in Pismo File Mount is probably possible. It is not an AVFS specific problem.

[Adub]

So, wait a second. Could I use this to create an avs script on one computer, have another linux computer connect to that script via network share, and then use my linux computer to encode that script? Is all avs processing done on the host computer? Like am I feeding raw data to my linux computer?

I am thinking that I might be able to apply this so that I run all of my scripting on an Avisynth and AVFS enabled XP machine, and then via a gigabit network, send the raw data a 64-bit Gentoo install were I encode the final output using mplayer with x264.

[joe48]

I am thinking that I might be able to apply this so that I run all of my scripting on an Avisynth and AVFS enabled XP machine, and then via a gigabit network, send the raw data a 64-bit Gentoo install were I encode the final output using mplayer with x264.


This should work. Post your results if you try it. Most interesting is how much CPU utilization you are able to get on each machine. Ideally at least one of the machines would have one CPU core fully utilized.

[Adub]

Cool. I can't do it yet, but I might be able to get it working this summer. If I do (or don't) get it working, I will be sure to post back.

[Adub]

Okay!! I got it working!! Which is good. However, it's disappointingly slow.

Here is what I did:

Log into windows, and then share my working folder.

I mount that shared folder in my linux machine with "sudo mount -t smbfs //ip.add.res.s/Videos /media/share"

I can browse around the shared folder and play the files using mplayer. I can even encode them!! Now, here is where the example comes in.

If I take my example file, and stream it directly to x264 via mencoder, I can get around 122fps on the first pass and ~12.18 fps on the second pass.

However, if I use AVFS, I get around 15fps on the first pass. I didn't even test the second pass.

The script was very simple, just a basic "Directshowsource("example.mkv")". I don't know if there is something I can do to improve it, but right now it's too slow for me to use in my encoding pipeline.

Edit: And as to CPU utilization, I didn't get above 20 % on either machine. The one running AVFS barely went above 10.

[Shinigami-Sama]

betcha your network maxed out
SMB is stupidly inefficient...

[Adub]

But, that makes no sense. Using the same network and same mounting techniques, I got 122 fps without using AVFS to 15 fps using AVFS. That makes little sense.

Unless.....Oh!!! Is it because I am sending UNCOMPRESSED? And that would result in bigger frame sizes, thus resulting in the decreased network speed, thus the decreased encoding speed. Is there a better alternative to SMB?

Note: This was on a 100mb network. Maybe if I can't fix it with an alternative to SMB, would an upgrade to 1000mb network work?

[mikeytown2]

Okay!! I got it working!! Which is good. However, it's disappointingly slow.

Since the network seems to be the bottleneck, make AVS the bottleneck. What i mean is, this would be useful when the AVS script takes 100% cpu, because then the network would no longer slow you down. So the idea of using this would make since when using monster scripts.

Also for alt to SMB i found this thread
http://www.neowin.net/forum/index.php?showtopic=308559

Edit: look at the taskman networking tab to make sure the network is maxing out

Edit2: If your looking at 10GB net (http://www.google.com/products?q=dual-port+10Gb+network&oe=utf-8&scoring=p), its about 400 bucks per computer...

[joe48]

However, if I use AVFS, I get around 15fps on the first pass. I didn't even test the second pass.


RGB24 AVI at 720x480 is ~1MB per frame. 80% of 100Mbit network is ~8fps. 30% of 1Gbit network is ~30fps. The frame rates double for YV12. Depending on your network, 15fps may be all you can hope for. The 2nd pass may run at the same speed though.
(Hope I got the math right)

[Shinigami-Sama]

RGB24 AVI at 720x480 is ~1MB per frame. 80% of 100Mbit network is ~8fps. 30% of 1Gbit network is ~30fps. The frame rates double for YV12. Depending on your network, 15fps may be all you can hope for. The 2nd pass may run at the same speed though.
(Hope I got the math right)

maybe passing it off to an i-frame only lossless codec could speed it up?
huffman maybe
should double or so the network access
shouldn't be to hard to do either

[mikeytown2]

maybe passing it off to an i-frame only lossless codec could speed it up?
huffman maybe
should double or so the network access
shouldn't be to hard to do either

VLC (http://www.videolan.org/) sounds like it could do the job. Speaking of VLC, could it accept avs files for input (http://forum.videolan.org/viewtopic.php?f=14&t=38283)?

[Adub]

Okay, so it IS my network. Let me see if I can find a gigabit card for my laptop.

[Shinigami-Sama]

actually
cold the fake AVI file be passed off to a http server? I don't see why not
you could easily have a script pass it off to a lossless encoder
though I guess that might not work for some applications...

[IanB]

People need to consider the data being virtualized here is big, really big. A RGB24 1920x1080 24fps progressive stream is 149,299,200 bytes per second. Even a YV12 (4:2:0) 720x480 60i is still 15,552,000 bytes per second. This is 24% more than the theoretical maximum transfer of a 100Mb/s ethernet (probably more than twice the practical throughput). These numbers are even a small chunk out of a 800MHz FSB memory channel at around 6,400,000,000 bytes per second.

[joe48]

Lossless compression in AVFS is not a solution to network bandwidth problems. AVFS would need to preprocess (compress) all frames to figure out file layout and indexing. It would make more sense to skip AVFS and losslessly encode to a real AVI file.

Serving over a network is still useful when the CPU overhead of the script and encoder are limiting throughput below what the network can handle.

[Adub]

Yeah, this is making more and more sense. Looks like I have a nice project to work on over the summer! I just need to setup a decent gigabit network, and look at my filter chain to see if I can change the bottleneck to be the avs, not my network.

[Zarxrax]

I've found a very strange bug. I have a folder full of a few hundred avisynth scripts, and all of them contain the line
AssumeFPS("ntsc_double")
I was going through mounting each one, one at a time, and i noticed that some of them would mount fine, but when i tried to mount others, it would crash. On these that failed to mount, I discovered that I could get them to mount by simply removing the AssumeFPS line. That's really strange though, since all of my scripts have that line, and some of them work. I've determined that it is this specific line that it causing it to crash though, in many of my scripts.

[joe48]

I've found a very strange bug. I have a folder full of a few hundred avisynth scripts, and all of them contain the line
AssumeFPS("ntsc_double")


If you can get me a sample exhibiting the problem I can look at it. Send me a PM.

[morsa]

Could something simillar to "FireHose" from Cinelerra be modified for our uses?

[Quark.Fusion]

About compression on network: did someone tried open share over compressed vpn link?

[Quark.Fusion]

Also about networking with avisynth: avisynth already have networking functionality with TcpServer(), will it work with mounted file?

What about adding multithreaded Lagarith compression for networking to AVFS? For user this can look like adding two variables to avs file with port number and compression method (like in TcpServer), on other side user will mount special file, that will re-create avi file in virtual filesystem (like using TcpSource). Of course all functionality already here (you can just mount TcpSource() avs file), but it is with ugly code (i was read that somewhere) and without lagarith compression.

[joe48]

I've found a very strange bug. I have a folder full of a few hundred avisynth scripts, and all of them contain the line
AssumeFPS("ntsc_double")

If you can get me a sample exhibiting the problem I can look at it. Send me a PM.


The bug Zarxrax found is a divide-by-zero with clips that are less than 1 second long. I will have the fix in the next release.

[joe48]

What about adding multithreaded Lagarith compression for networking to AVFS? For user this can look like adding two variables to avs file with port number and compression method (like in TcpServer), on other side user will mount special file, that will

TcpSource and AVFS should interoperate fine.

I am not very familiar with TcpSource. It should not be necessary to know the compressed size of each frame in advance, so lossless compression could be implemented...

[tally3]

TGA sequences would be really useful!
http://www.organicbit.com/closecombat/formats/tga.html

[thommyfilm]

hello! sounds like very good work. one (maybe dumb) question: you say you would add output file format options. would it be possible to show an mpeg4 h264 file on the filesystem instead of a avi file? this should happen without reencoding. the input files are already mp4 h264 files.

[krieger2005]

You can't do it without reencoding. Every change, which you made with an avisynth-script to your video must be reencoded. This is a simple Law:
-> No change -> No Reencoding -> You don't need to use avisynth
-> change (in anyway) -> need reencoding

[mikeytown2]

@thommyfilm, thats what MeGUI is for ;)

[Adub]

Uh, guys, I think he is asking if support for a "fake" mp4 front could be added in addition to the "fake" avi front. I don't know why this could be, as everything supports avi, while mp4 is much more limited.

[IanB]

Also would programs that accept MP4 containers handle uncompressed video and audio data. I suspect not.

[joe48]

There may be some performance changes when AVFS is used with the latest PFM Audit Package (build 048). I changed the caching in the file system driver to use 256k blocks instead of 48k. I did not test this against AVFS, probably should have...

I have some optimizations in the works for Pismo File Mount that will improve AVFS performance. Mainly there will be buffering improvements to eliminate extra data copying. It will be a while before I get this out, probably January.

Joe L.

[Adub]

Great!! I look forward to the next release!

Thanks for your hard work joe48.

[hanfrunz]

I have another wish. Is it possible to mount multiple .avs to one virtual folder or drive? So lets say i have clip1.avs, clip2.avs and i like to have \clips\clip1.avi and \clips\clip2.avi? So i could import two files via drag and drop to my encoder.

that would be realy helpful.

regards,
hanfrunz

[joe48]

I have another wish. Is it possible to mount multiple .avs to one virtual folder or drive? So lets say i have clip1.avs, clip2.avs and i like to have \clips\clip1.avi and \clips\clip2.avi?


On vista you might be able to fake this using a file search.

If the \clips\ folder is a real folder then this would be difficult. On vista it could maybe be done by having avfs create and delete NTFS symbolic links.

If the \clips\ folder is a virtual folder, then it would require more UI, probably an .avs file list mount file. It also would result in multiple avisynth instances running in a single process, which might be problematic.

[hanfrunz]

On vista you might be able to fake this using a file search.

filesearch is a very good idea and should work on xp as well, thanks!

[mikeytown2]

Any chance of adding YUV output as well as RGB?
http://forum.doom9.org/showthread.php?p=1218241#post1218241

[IanB]

@mikeytown2,

:o :o It already does. :o :o

AVFS outputs the exit format of your Avisynth script, RGB24, RGB32, YUY2 or YV12

[paulvdb]

I can't get this to work. I download and installed Pismo and then registered the avfs dll. When I mount an avs file it creates a new folder with that same avs file in it and a file called error.log which just context the text "(null)". Should I do anything else to get it to work? I've tried this with several avs files including the sample.avs that came with AVFS. I'm using Windows XP SP2 with all the latest updates.

[paulvdb]

I managed to solve my problem by upgrading avisynth. Apparently AVFS doesn't work with avisynth 2.56.

[joe48]

I released a new version of Pismo File Mount Audit Package earlier this month, build 050.

This new version has some substantial changes to buffering and caching to eliminate unnecessary memory copying. Depending on data sources and AVS scripts, there might be some performance increase using AVFS on build 050 and later. No changes to AVFS itself were necessary, so the AVFS build has not been updated.

Build 050 also uses 128kb sized cache buffers in the driver, instead of 256k in build 047-048, and 48k in build 046 and earlier. This only affects client applications that do not do unbuffered I/O.

Joe L.

[Quant]

And how about BoxedApp SDK (http://boxedapp.com/). I heared, that this SDK is for emulating file system and registry too? Does anybody use this?

[a451guy451]

I love this thing, but it only lets me host virtual files (that my Mac can see) on shared folders physically on the host windoz machine. Is there a way to get files on independent network storage to work, because while SD sourced files work great HD stuff chokes on my ethernet card? Right now, I technically can make virtual files on network storage, but they are only accessible as such via my host machine which defeats the purpose. Or perhaps this is impossible and I need to go buy a 10gbps ethernet card... Thanks for any input!

[a451guy451]

Oops. I think I'm having a FSB bottleneck, not ethernet. Just swapped my penrin host machine with an i7 and it dramatically improved.

Oddly enough, if I connect to my virtual folder from any PC on my network, everything works great. It seems my Mac is having a personal problem (or perhaps quicktime is specifically). Does anybody else make virtualfiles using HD sources and use them on their Mac? I'm low on apple expertise, and am not really sure how to troubleshoot the system for the bottleneck there.

[Dreadkillz]

Hi, I would really love to use this, but whenever I so much as select the avi file from my script, my folder explorer would just freeze. I checked that pfmhost also grows in memory usage by a lot. Loading the virtual file is A LOT slower than loading the actual script file for me. Also, the heavier that my filters are, the slower it is to load. BTW, I have an XP with AMD 3600 x2

[joe48]

Hi, I would really love to use this, but whenever I so much as select the avi file from my script, my folder explorer would just freeze. I checked that pfmhost also grows in memory usage by a lot. Loading the virtual file is A LOT slower than loading the actual script file for me. Also, the heavier that my filters are, the slower it is to load. BTW, I have an XP with AMD 3600 x2

Performance is often slower with AVFS than a frame served or VFW accessed AVS, but it depends on the script and on the program accessing the virtual AVI file. The problem is not that AVFS adds any significant overhead to rendering, because it doesn't (there are file systems built on PFM doing hundreds of MBs per second sustained throughput). The problem is that the programs accessing the virtual AVI files often issue reads that require frames to be rendered that the application does not use.

With the limited types of scripts I use, the performance issue I see is a delay when opening the virtual AVI. This is caused by sloppy reads of the AVI file headers and indexes inadvertently causing unneeded frames to be rendered.

There may be some things that can be done with the tunable AVFS parameters in the script to improve performance. Also, someone with some C/C++ coding experience, who is seeing performance issues, could experiment with caching and/or padding in a custom avfs.dll.

[IanB]

... but whenever I so much as select the avi file from my script, my folder explorer would just freeze. I checked that pfmhost also grows in memory usage by a lot. ...
Various tools and explorer plugins like to 'investigate' .AVI files. And then there are virus scanners and security products. It is crucial these never be allowed to read through the contents of the pseudo .AVI files. If they do then the pseudo contents has to be rendered. If it takes 2 hours to render the full Avisynth script, then it will take 2 hours to read through the pseudo .AVI file.

Joe, does PFM have knowledge of the "Application name" when a pseudo file access happens? Could various applications be black listed and the returned contents just be 1 big zero fill. Or maybe a white list so only say VirtualDub.exe, X264.exe and Vegas*.exe get to see the rendered pseudo contents.
... The problem is that the programs accessing the virtual AVI files often issue reads that require frames to be rendered that the application does not use.

... This is caused by sloppy reads of the AVI file headers and indexes inadvertently causing unneeded frames to be rendered. ...
In the original implementation, I went for up tight maximum packing in the pseudo AVI file. On reflection perhaps a sparse highly padded approach would have been better.

In an AVI structure 'JUNK' segments can be freely used to align/pad other data segments, this is normally done when capturing uncompressed video to match the clustering alignment of physical disks and maximise performance.

It would be trivial to have the frame data aligned on some nice round offset (say 64K or more) and add a 'JUNK' padding/guard block between each frame chunk. The data contents in the 'JUNK' blocks doesn't even matter, so no actual memory transfer/zero fill needs to occur, further boosting performance.

I assume the marshalling code is happy if no actual data is written to the application's buffer.
_____________________________________

Case study :- On mapped Linux systems the /dev/zero driver can apparently achieve throughputs of 100's of GB/s, way in excess of the actual memory bandwidth of the host CPU. This is achieved by zero filling 1 physical memory page and mapping it repeatedly onto the users buffer. E.g. App does a 1Mb _read of /dev/zero, on a modern CPU one might expect this to take 40 microseconds (25GB/s), profile the _read call and it only takes 5 microseconds (200Gb/s). Woooohoooo!

[joe48]

Various tools and explorer plugins like to 'investigate' .AVI files. And then there are virus scanners and security products. It is crucial these never be allowed to read through the contents of the pseudo .AVI files.

.....

Joe, does PFM have knowledge of the "Application name" when a pseudo file access happens? Could various applications be black listed and the returned contents just be 1 big zero fill. Or maybe a white list so only say VirtualDub.exe, X264.exe and Vegas*.exe get to see the rendered pseudo contents.


No current mechanisms exist in PFM to allow the formatter to change behavior based on the process name. If it were supported, it would only be available during open, not at read.

But, I really do not like this sort of a solution. It looks like a quick fix, but it has a lot of drawbacks.

Examples:
1) Explorer related I/O, such as various shell extensions, is going to happen in the open dialog in applications in the white list.
2) Some of the anti-virus products and system extensions open files and do I/O in the context of the triggering process (this is true of prefetch as well).
3) Some apps/plugins may do work out-of-process, so the user has to track down the name of processes they did not run and add them to the white list.
4) Dllhost.exe, rundll32.exe, and the like.

One thing that should help is to set the FILE_ATTRIBUTE_OFFLINE flag on the virtual media files. This is a trivial change to avfs.dll. I would have done this originally, but it was not supported two years ago when we first put AVFS together. This flag will convince explorer to leave the file alone, only show default icon, etc.. Many 3rd party shell extensions and applications also honor this flag, otherwise they are unusable in enterprise environments that use any sort of HSM. The shell extensions that do not check this flag should be bugged back to the developer.

The anti-virus issues can be handled (usually) with exclusions. Well, if the anti-virus product does not support user exclusions then the user might want to look for another vendor.

In the original implementation, I went for up tight maximum packing in the pseudo AVI file. On reflection perhaps a sparse highly padded approach would have been better.

...

It would be trivial to have the frame data aligned on some nice round offset (say 64K or more) and add a 'JUNK' padding/guard block between each frame chunk. The data contents in the 'JUNK' blocks doesn't even matter, so no actual memory transfer/zero fill needs to occur, further boosting performance.


I can not recall if there were any compat issues with the padding between frame and audio chunks. There is the existing AVFS_AVI_NoInterleave option to disable interleaving the audio and video chunks. Some more padding options would probably be useful.

The below thread indicates the no-interleave option solves some performance issues. I am suspicious about the comment that it introduces audio sync issues, but this is the sort of compat trouble that playing with the index and with padding is likely to cause.
http://www.dvinfo.net/forum/canon-eos-5d-mk-ii-hd/140473-intermediate-codecs-windows-users-2.html#post990122

[Robert Martens]

Talk about a small world; after years of lurking, I registered an account here a few weeks ago to post about an Avisynth helper script I'd developed. I ended up revising it practically every other day, though, and thought it best to wait until I'd settled on something more stable before bringing it up here, so I have yet to get around to that (maybe later tonight, actually), but I was just browsing the boards and stumbled across this.

As evidenced by my username, I'm the same guy from the aforementioned DV Info Net thread. It's been over a year since I last experimented with the AVFS as it applies to 5D Mark II Quicktime files, so I'm a little fuzzy on the details, but yes, I do remember that while there was a performance improvement brought about by AVFS_AVI_NoInterleave, we subsequently ran into an issue with A/V sync. Barely being able to "Hello, World!" my way out of a wet paper bag, I was and still am in no position to diagnose the cause, and the simplest solution was just avoiding that option. Ultimately I believe most people started using a Cineform transcoding solution for their 5D (and now 7D and T2i) footage for use in their NLEs.

I still have the sample clip I was using for tests in that DVi thread (on the same system, no less, with the same processor, RAM and hard drives, though now I'm using Avisynth 2.5.8 MT), and I'm happy to help if there's anything I can do as far as testing. Or, at least, shedding some light on that particular discussion. It's been awhile, but I'm sure I could remember the details if you have any questions.

[joe48]

I do remember that while there was a performance improvement brought about by AVFS_AVI_NoInterleave, we subsequently ran into an issue with A/V sync.
....
and I'm happy to help if there's anything I can do as far as testing. Or, at least, shedding some light on that particular discussion. It's been awhile, but I'm sure I could remember the details if you have any questions.

If you get a chance, could you rerun the test with and without the no-interleave option and verify the audio sync issue comes and goes?

[Robert Martens]

Turns out more had changed than I remembered, in that I had been using the CoreAVC decoder and loading the fake AVI files into a trial copy of Vegas Pro. I've downloaded Vegas too many times now (I've needed it more than once to help someone diagnose a problem; it now tells me the trial period has expired, so I can't register the download), and pathetic as it may seem I can't throw any money at CoreAVC for the moment.

I have nonetheless confirmed the behavior in question by other means. I have ffdshow-tryouts version 3154 installed, and I have the H.264/AVC "decoder" dropdown set to ffmpeg-mt for both DShow and VfW, since I find that performs best on my system.

I created a simple script, AVFSsynctest.avs, containing these lines:

AVFS_AVI_NoInterleave=false
DirectShowSource("C:\Downloads\mvi_1419.mov")
ConvertToRGB24(matrix="PC.709")

The source clip doesn't appear to be in the same place as when I grabbed it, but that Vimeo user referenced in the DVi thread has made the clip available here: http://drop.io/larktav

I swapped NoInterleave between true and false, and commented out the ConvertToRGB line for a couple of tests. The script was mounted with the latest PFM/AVFS, the files were opened in Liquid 7.2, combustion 3.0.4, and MediaInfo, and the script was explicitly Unmounted between each change. Results were as follows:

1.) NoInterleave=false->DirectShowSource->ConvertToRGB24->Liquid 7.2

On a 1080/30p timeline (exactly 30p, not 29.97), everything displays as if it's fine; the audio in the AVI file shows up as being the same duration as the video, as do the two WAV files generated (AVFSsynctest.wav and AVFSsynctest.00.wav). I'm inclined to believe this is a quirk of the NLE, however, thanks to tests 3 through 8.

2.) NoInterleave=true->DirectShowSource->ConvertToRGB24->Liquid 7.2

Same as 1; everything looks fine, durations all appear to match.

3.) NoInterleave=false->DirectShowSource->combustion 3.0.4

Here's where the first sign of trouble begins. The video track is reported as 491 frames, but audio is reported as 490 whether I load it from the AVI container or either of the WAVs.

4.) NoInterleave=true->DirectShowSource->combustion 3.0.4

Even stranger; the durations behave as in test 3, but trying to load the AVI as an audio source causes only the last ten frames or so to display a waveform in the combustion interface; further, the audio track is silent even during those ten frames. The WAVs as audio sources had the 490 duration, but played back without issue.

5.) NoInterleave=false->DirectShowSource->ConvertToRGB24->combustion 3.0.4

For tests 3 and 4 I had initially removed the ConvertToRGB line, as combustion doesn't require it, but thinking it might have some impact on my results I restored the line and tried again. Results were identical to test 3.

6.) NoInterleave=true->DirectShowSource->ConvertToRGB24->combustion 3.0.4

Same as number 4.

7.) NoInterleave=false->DirectShowSource->ConvertToRGB24->MediaInfo

Finally, I loaded the phantom files in MediaInfo 0.7.19 to see what it could tell me, and found that in this case the AVI video track showed a duration of 16s 366ms, and the audio a duration of 16s 333ms. The external files, AVFSsynctest.wav and AVFSsynctest.00.wav, were also both 16s 333ms.

8.) NoInterleave=true->DirectShowSource->ConvertToRGB24->MediaInfo

Changing only the NoInterleave option, this time I found the AVI video track was its expected 16s 366ms, but the audio packed into the AVI had shrunk to 15s 867ms. AVFSsynctest.wav and AVFSsynctest.00.wav, though, were both 16s 333ms, as I'd seen in the other examinations.

All of this is in line with what we experienced in that thread; I was unclear at first, but as soon as I saw those numbers in MediaInfo I remembered running into the same audio-shrink last time.

[IanB]

Joe, Seems you have a bug, you forget to put back the 0.5 secs of preload when not in interleave mode.

[joe48]

@Robert, I sent you a PM with a link to a test build of AVFS. It would be great if you could give it a look.

[joe48]

As Ian noted, AVFS is truncating .5s off the audio stream when the no-interleave option was set, creating an invalid AVI file. I put together a fix for this and Robert verified it resolved this issue. The fix will be in 1.0.0.5, which should be out pretty quickly.

The extra frame in the virtual AVI compared to the source file (491/490, 16.366s vs 16.333s) seems related to the DirectShowSource and ffdshow-tryout setup. Using FfmpegSource the frame count and duration are correct, verified by Robert. I have given up on reproducing this as I have been unable to get DirectShowSource and ffdshow-tryout to work on my system, a 2 yr old Vista 64 install.

The "open as audio only" issues are probably limitation of the specific application. I expect most applications will have trouble with non-interleaved audio, that was my experience during the initial AVFS development.

Thanks for the help Robert.

[Robert Martens]

You're welcome, I'm glad to be of service.

I think I've finally learned my lesson, and I'll stick to FFmpegSource and DGDecode for my non-AVI files from now on. Loading the clip as audio-only is likewise something I don't foresee a need for, interleaved or not, so if I need an external file as an audio source I can simply use the WAV files AVFS creates.

Thank you for the fix, I'll be sure to append that DV Info Net thread with the information once 1.0.0.5 is released!

[joe48]

I know a few pople have worked out how to setup scripts so AVFS will work with specific products, such as Quicktime and Final Cut Pro. In particular, being able to import without converting to RGB.

I would like to get this info in the AVFS readme to save some users having to figure this out, or giving up trying. Please post any tips you think will help.

Also, anyone who has had problems with things like explorer, shell extensions, anti-virus, content indexer, etc., causing grief by reading the virtual AVI files, get that info posted here as well so I can look into it.

[JoeH]

Hi Joe!

Thanks for this product. I use it together with DGIndexNV in order to import H264 / VC1 files directly into Sony Vegas Pro 8 for editing. I find that final rendering is about 25% faster than this way compared to using an intermediate codec, and obviously it's great for disk space.

The only difficulty I have found is that the thumbnails in Vegas are painted very slowly - it takes about 10 seconds usually to paint the whole timeline of thumbnails. In comparison, if I use an intermediate codec it takes around 1 second.

I have an NVidia 9500 GT which is reading the video using DGIndexNV. As Vegas can't import AVS scripts, I use your AVFS to create an AVI which I import. Any ideas for getting improved performance on the thumbnail creation?

[joe48]

Hi Joe!
The only difficulty I have found is that the thumbnails in Vegas are painted very slowly - it takes about 10 seconds usually to paint the whole timeline of thumbnails. In comparison, if I use an intermediate codec it takes around 1 second.
....
Any ideas for getting improved performance on the thumbnail creation?

I do not have any experience with Vegas (or any other pro editing tools for that matter), so do not know any specifics. Things are going to be slower with NLE tools since you have the script overhead for every frame that is touched by a read from the editing app.

You may be able to setup your script so that you can easily disable/enable the more time consuming processing steps. Use the script with the time consuming steps disabled when doing editing, then enable full processing and reopen your project before generating final output.

[Jim_Pansen]

I found the AVFS today & I'm very impressed! It matchs my needs for using AVS scripts in my workflow.
It speeds up my workflow immediately!
THX a lot for developing this!

Great tool!

Jim

[kolak]

Is there a way to implement V210 at the output, so in case of 10bit codecs (v210, Cineform, ProRes, DNxHD) we could keep this precision?
It's a great and simple piece of software :)

Thanks,
Andrew

[IanB]

@kolak,

Avisynth does not have any internal pixel formats other than 8 bit at present.

And the design here is to output uncompressed pixel data wrapped as an uncompressed virtual .AVI file.

This is an open source project, sure you could add a compression interface, but why would one want to burn cpu cycles generating pixel data. then compressing it into memory, only to then decompress it again immediately to use the raw pixel data. :confused:

[kolak]

@kolak,

Avisynth does not have any internal pixel formats other than 8 bit at present.

And the design here is to output uncompressed pixel data wrapped as an uncompressed virtual .AVI file.

This is an open source project, sure you could add a compression interface, but why would one want to burn cpu cycles generating pixel data. then compressing it into memory, only to then decompress it again immediately to use the raw pixel data. :confused:

You did not understand me correctly. I don't want to put any codec in between- these are exapmle of 10bit codecs only.

I just would like to keep 10bit precision. If my source file is 10bit (eg Cineform file) I would like to have 10bit uncompressed virtual file. It gets converter to YUY2 now. This is what I expcted, it's an avisynth limitation :(

Is there any trick to pass v210 output from eg. Cineform, ProRes?


Andrew

[StainlessS]

WARNING, Pismo File Mount Audit Package (pfmap-159-win.exe) is flagged
by COMODO AntiVirus as suspicious (Today with just updated AV DB).
Have submitted to VirusTotal.com where only COMODO out of 42 AV scanners
had any concern over it. Have submitted to COMODO as false +ve.

[joe48]

WARNING, Pismo File Mount Audit Package (pfmap-159-win.exe) is flagged
by COMODO AntiVirus as suspicious (Today with just updated AV

Anti virus products should be required by law to have a warning label attached to their packaging.
+------------------------------------------
| WARNING: Windows security products
| contain snake-oil, a substance known to
| the admin general to deteriorate the
| performance, reliability, and security of
| computer systems.
+------------------------------------------
(joke)

This looks like a heuristic (fuzzy logic) detection. Likely the anti-virus product noticed that the PFM installer loads a kernel driver. Possible the anti-virus product mistakenly thinks the PFM installer is a packed executable. I doubt PFM has specifically been classified as malware by the anti-virus product.

The PFM install package and every contained executable/DLL is signed with a code signing certificate. Most security products trust signed executables until they get a sample of some malicious code that was signed with the same certificate. I am surprised if Comodo is not doing the same.

Joe L.

[StainlessS]

Yes, just suspicious, as Heur.Suspicious@116614127.

Also, as an aside, I think Comodo is supposed to be one of the
biggest certificate issuers in the world.

[JoeH]

The only difficulty I have found is that the thumbnails in Vegas are painted very slowly - it takes about 10 seconds usually to paint the whole timeline of thumbnails. In comparison, if I use an intermediate codec it takes around 1 second.


I found a solution to this problem - index the file with FFMS2 and then mount with AVFS the AVS script that opens the ffindex file. The H264 / VC1 files fly... just like an intermediate codec. Very useful if you need to edit one in Sony Vegas, or any other video editing program.

[name123]

i like this concept but i am unable to test this out because when i run the installer for pismo film mount it crashes my win7 x64, has anyone experienced this?

[joe48]

i like this concept but i am unable to test this out because when i run the installer for pismo film mount it crashes my win7 x64, has anyone experienced this?

PFM runs fine on Win7 x64. A crash is most likely a compatibility issue between the PFM kernel extension and some other kernel extension or malware installed on the system. I am not aware of any specific compatibility issues at this time, but there have been a few cases over the last few years. If you are interested in helping to diagnose the issue, contact me via the support e-mail address listed on the Pismo Technic website.

[name123]

will do, thanks

[torwart]

You are a genious joe48!!!! Great program!!!!!!

[JoeH]

I use it on Win7 64 without any problems.

[Overdrive80]

Hi guys, i have a problem when i try reg dll with pfm command but windows says: "Not recognized as internal or external command, program or file batch executables".

Any idea?

[joe48]

Hi guys, i have a problem when i try reg dll with pfm command but windows says: "Not recognized as internal or external command, program or file batch executables".


Sounds like the PFM Audit Package is not installed.

[Overdrive80]

I try install "Pismo File Mount Audit Package build 159" but merge windows with this message:

http://www.imagengratis.org/images/capturadj7td.png

[joe48]

I try install "Pismo File Mount Audit Package build 159" but merge windows with this message:


PFM installs and functions on the vast majority of Windows configurations. If you are interested in helping to diagnose the issue with your configuration and PFM, contact the support e-mail address listed on the Pismo Technic website.

[verydoomed]

I am confused, I have the Pismo File Mount system extension and it works fine, why do I need this AVFS thingy? what is the advantage?

[IanB]

The Avisynth Virtual File System is a Pismo File Mount system extension to mount an Avisynth script as a virtual .AVI file. The .AVI file can then be opened in various programs that cannot normally open Avisynth scripts.

[Mug Funky]

quick question...

i've had some luck serving to crapple land by opening the yuy2 avi file, saving a reference mov of it, then using a text editor manually searchreplacing "yuy2" with "yuvs".

i just did some googling, and found this link:

http://developer.apple.com/quicktime/icefloe/dispatch020.html

this is telling:

kYUVSPixelFormat
'yuvs'

8-bit 4:2:2 Component Y’CbCr format. Identical to the k2vuyPixelFormat except each 16 bit word has been byte swapped. This results in a component ordering of; Y0, Cb, Y1, Cr. This is most prevalent yuv 4:2:2 format on both Mac and Win32 platforms. The equivalent Microsoft fourCC is ‘YUY2’.


now i might add that there seems to be some colour encoding problems - when i tried going SD to HD a couple of years ago, i found the colours shifted. but what may have happened is FCP assumed 601 coeffs even though i gave it 709 data.

anyway, if yuy2 is simply a byte-swap of 2vuy, it should be trivial for AVFS to just do a byte-swap and make the mounted avi file a "2vuy" (blackmagic or other SDI capture card, 8-bit 4:2:2) instead of a yuy2.

this would immediately remove a lot of manual text editing and possible colour mismatches from anyone trying to get avs content smoothly into quicktime land...

[IanB]

Setting the fourcc in the pseudo .avi header to "yuvs" is zero cost. Other fourcc's with the same byte order as 'YUY2' include 'yuyv', 'YUYV' & 'YUNV'.

Doing a word swap to implement '2vuy' would require additional processing. Currently VFB data is simply copied to the application buffer. Other fourcc's with word swap byte order of 'YUY2' include 'UYVY', 'UYNV', 'cyuv', 'IYUV', 'HDYC', 'Y422' & '2vuy'.

[Mug Funky]

that's good to know.

the reason i was looking at 2vuy is because there's a limited number of formats that final cut pro is willing to play out through an SDI interface without having to render anything.

though network bandwidth would most likely mean having to render anyway, i thought having AVFS do the grunt work would possibly allow a fast enough network to go to tape in realtime straight from an avs script.

my test last night showed no colour shifts, so i'm gonna go ahead and check HD now.

[k-c-ksum]

I try install "Pismo File Mount Audit Package build 159" but merge windows with this message:

http://www.imagengratis.org/images/capturadj7td.png

reboot then install is as "Run it as Administrator"

[tateu]

Is anyone else seeing unbearably huge memory leaks?

My system:
Windows 7 Pro x64
2 X5550 Intel Xeon Processors @ 2.79 GHz
16 GB Ram
AviSynth v2.58 tsp MT v5 2009/07/12 07:46:21
Pismo File Mount Audit Package build 159 (1.0.0.159.x64 2010/07/07)
AVFS 1.0.0.5 2010/05/23

After a clean reboot my System process is only using a few hundred KB of memory.

AviSynth Script used
ColorBars(928,768).Trim(0,1799)
or
AviSource("lagarith_no_audio_928x768.avi").Trim(0,1799)

pfm mount test.avs
Open mounted test.avi file in VirtualDub or the player/encoder of your choice.

In VirtualDub, press F5 to play the avi. By the time all 1800 frames have been played, my System process is using over 2GB of memory. If I close and reopen the file, it will push my System process memory usage up to a max of 4GB.

Closing and unmounting the file does not release the memory. The only way I can get it back is to reboot.

[tateu]

After more investigation, it appears that the memory leak might be with the Pismo File Mount Audit Package itself, not just the Avisynth virtual file system plugin.

I created an uncompressed avi, zipped it and mounted the zip file with the Pismo File Mount Audit Package zip handler and then opened the mounted avi in VirtualDub. I experienced the same memory leak in the System process with the mounted zip file as I did with the mounted avs file: within a short time of playing the avi file my System process had consumed 1/4 of my total installed memory and it doesn't get released until I reboot. I have tested this on several different machines and two different OSes.

Pismo Technic Inc. has a support email address so I will send them a note about it and see if they have anything to say.

[joe48]

Is anyone else seeing unbearably huge memory leaks?
...
After more investigation, it appears that the memory leak might be with the Pismo File Mount Audit Package itself, not just the Avisynth virtual file system plugin.


PFM does not have a memory leak. The number showing in task manager is just a number, there are no other symptoms to bear. If you had a huge memory leak then your system would get sluggish (I mean more than normal for Windows), and would eventually crash.

Most file systems on Windows use ram to cache data, true of PFM and NTFS. The amount of ram used by PFM to cache data is under complete control of the system (Windows), the same as for NTFS. For technical reasons, PFM maps its cached data differently than other file systems resulting in the current ram usage showing up in task manager in the system process. The cache ram usage of NTFS does not show up in the system process because it maps its cached data in the kernel, not because any more or less ram is being used.

The large number you are seeing in task manager is because you have plenty of ram installed in your machine and the system has not yet had any need to reallocate ram from the PFM cache to other tasks. As soon as you run some other memory hungry tasks, the system will reallocate the ram for other uses.

Joe L.

[ffffuuu...]

Hi there,
I've been aiming to use NTFS (under XP) to open a video from DVD structure in an app that only handles avi, so my goal was to save as much space (on mobile HD) as possible. I extracted audio tracks while indexing with DGMPEGDec, and then wrote a script using mpeg2source for the d2v index file, alternatively NicAC3source or FFAudioSource for ac3 file and AudioDub for joining the two. The resulting script played well in Media Player Classic, but the avi file created with AVFS rapidly slowed down to about 10fps and went out of sync. A video-only avi created with a script using only mpeg2source played at more less right framerate. What problem may there be?